Project Summary Electronic health records contain a wealth of information about patient health status that can be mined for multiple purposes, including clinical research and improved decision-making at the point of care. This information can be represented as structured variables, unstructured text, and images, among other data types. In this work, we develop new models for representing the unstructured text that take advantage of powerful neural models called pre-trained transformers. We propose to make these models usable for much longer texts by adding hierarchical layers to operate over summaries of smaller chunks of text, and shrinking the size of the encoder that operates on smaller chunks. First, we develop a smaller encoder for sentence and paragraph-sized texts, by using a technique called extreme distillation that trains smaller models from the output of larger models. We also propose to pre-train hierarchical models for text, by taking advantage of smaller encoders like that from the first aim. We take advantage of both public and private datasets and experiment with different pre-training tasks and architectures. Our final aim proposes to combine representations learned from text with those from the more mature areas of structured data and images. We design experiments that answer the question of how best to merge these different information sources, and apply them to important clinical classification use cases that are likely to require multiple information sources for accurate performance. Specifically, we address the clinical tasks of predicting injury severity in emergency departments, and predicting diagnosis and prognosis of patients in intensive care units.

Project Narrative The most accurate current methods for understanding text with computers have severe restrictions on the lengths of the text that they can process, which is a severe limitation for current systems’ understanding of electronic health records. In this project, we first propose new algorithms that can handle longer text sequences, along with innovative ways to learn the models. Our second major goal is to incorporate multiple data types in the same architecture, combining structured data from traditional databases, unstructured text, and images. We evaluate our proposed methods on several important clinical tasks, including predicting injury severity in patients in the emergency department, diagnoses in intensive care unit patients, and several prognostic variables in intensive care unit patients.