Pilot intervention studies (aka feasibility or preliminary studies) play an indispensable, fundamental role in the development, refinement, and dissemination of social science/public health interventions. In pilot studies, preliminary evidence on important processes and potential efficacy of an intervention are collected. Despite their prominence in the development and funding of almost every well-powered randomized intervention, pilot studies have received very little attention regarding how they should be designed so that the study’s findings will provide information to inform decisions about further testing and refining of an intervention. Other guidelines, such as CONSORT, TREND, SPIRIT, or TIDeR, focus on factors associated with internal validity or transparency/replication, and fail to address substantive issues interventionists need to consider during the early stages of testing an intervention, such as what is delivered, who its delivered to, and the intensity of support for delivery and whether these can be scaled in a larger, well-powered trial. Based on our preliminary findings, these issues, which we refer to as “generalizability biases,” are a few of the hypothesized emerging factors that lead to “false-positive or exaggerated early discoveries” and subsequent failed well-powered trials. The identification and avoidance of generalizability biases can guide intervention decisions during the early testing so that, according to the NIH, a pilot study’s results can “…enhance the probability of obtaining meaningful results in subsequent well-powered trials.” In the proposed study, we will conduct a comprehensive systematic review and meta-epidemiological assessment of the role pilot studies play in the testing and scaling of interventions. This information will be used to inform the field of scientific practice on how to design more informative pilot studies that can increase the probability of success of interventions in well-powered trials. For this study, we will use an innovative multi-phase, cross-validation approach to develop a working set of generalizability biases interventionists should avoid when conducting preliminary tests of an intervention. In Phase 1 we will identify, define, and catalogue candidate generalizability biases within a large body of pilot intervention studies (N = 740) on the topic of child obesity. In Phase 2 we will cross-validate the biases, refine them, and identify new ones in a sample of pilot intervention studies on the topic of adult obesity (anticipated sample of 800 studies). In Phase 3 we will perform a double cross-validation of the biases in a new sample of pilot interventions on the topics of tobacco and HIV/AIDS (anticipated sample of 400 studies each). In Phase 4, a working draft of the principles will be refined via Delphi survey with leading Journal Editors. This study is significant because it will be the first to systematically develop guidance for intervention pilot studies. This study is innovative because it will address a topic that has largely been ignored (pilot studies) and will synthesize both quantitative and qualitative data to understand how pilot studies can be improved – an essential, yet overlooked aspect of developing high-quality interventions.

Project Narrative Intervention pilot studies (aka formative, feasibility, proof of concept, evidentiary, or vanguard studies) play an indispensable, fundamental role in the development, refinement, and eventual dissemination of social science/public health interventions. Despite their prominence in the development of almost every large-scale randomized intervention trial, pilot studies have received very little attention in terms of how they should be conducted, what information should be collected and reported, and how results should be interpreted. This study will develop conduct and reporting guidelines to establish methodological rigor for the conduct, interpretation, and reporting of pilot studies designed to inform the testing of social science/public health behavioral interventions in fully powered randomized trials.