Improving conversational use of spoken language is an important goal for many new interventions and treatments for children with neurodevelopmental disorders. However, progress in testing these treatments is limited by the lack of informative outcome measures to indicate whether or not an intervention or treatment is having the desired effect on a child's conversational use of language (i.e., discourse skills). The long-term goal of the proposed renewal project is to harness the benefits of NLP to impact functional spoken language outcomes for children with neurodevelopmental disorders. The goal of the parent R01 (R01DC012033) is to develop and validate new Natural Language Processing (NLP) based methods that automatically measure discourse-related skills, including language productivity (talkativeness), grammar and vocabulary, and discourse, based on raw (i.e., not coded or annotated) transcripts of natural language samples. Our objective in this proposal is to take the next step to evaluate the suitability of these NLP-based measures as outcomes for children with a range of intellectual abilities, language levels, and diagnoses. NLP algorithms require choices of pivotal parameter settings, such as word frequency dependent weights. While our previous results, involving between-group contrasts, were insensitive to these settings, our proposed project, involving psychometric quantities such as validity, may be sensitive to them. Building on our progress from the parent R01, we propose to pursue three specific aims: (1) Identify pivotal parameter settings that optimize stability of NLP discourse measures, and examine responsiveness to real change; (2) Evaluate consistency of NLP discourse measures, and identify key measurement factors that impact consistency; and (3) Evaluate validity of NLP discourse measures, and differences in validity as a function of diagnostic group, age, IQ, and language ability. Our approach will focus on optimizing stability of such measures, and assessing responsiveness to change over time, consistency across sampling contexts and different sample lengths, and validity of each measure. The contribution of the proposed project will be to systematically assess the psychometric properties of NLP discourse measures. The proposed research is innovative because it represents a substantial departure from the status quo by taking the crucial next step: the development of scalable, psychometrically sound measures of discourse skills that can be used to assess between-group differences as well as within-individual change over time. The proposed research is significant because it is expected to result in viable spoken language outcome measures for children with a range of neurodevelopmental disorders, making it possible to target and meaningfully measure improvements in clinical trials and behavioral interventions. Ultimately, the successful completion of this study will provide the immediate ability to scale up treatment evaluations involving measurement of spoken language use, allowing flexible data collection across sites and studies, and in the future provide new targets for to-be-developed behavioral and pharmacological interventions.

The proposed research is relevant to public health because improving conversational use of spoken language is an important goal for many new interventions and treatments for children with neurodevelopmental disorders, and outcome measures that are automated, quantitative, scalable, and objective are needed to evaluate these treatments. The proposed research is relevant to the mission of NIH because it contributes to the development of fundamental knowledge about the spoken language use among children with a range of neurodevelopment disorders and conversational language difficulties, and will make it possible to target spoken language and meaningfully measure improvements in clinical trials and behavioral interventions.