Our team at the Prevention Research Center within the Brown School at Washington University in St. Louis, with guidance from experts at Kaiser Permanente Washington Health Research Institute, Drexel University Dornsife School of Public Health, and Brigham Young University School of Social Work, conducted a systematic literature review to:

  • Learn what quantitative measures are used to assess health policy implementation outcomes (e.g., acceptability, appropriateness, cost, feasibility) and determinants.
  • Rate pragmatic characteristics of those measures.
  • Assess their psychometric properties.
  • Identify measurement gaps.

This review focused on both implementation determinants and outcomes. Outcomes were defined according to Proctor and colleagues (2011): acceptability, adoption, appropriateness, cost, feasibility, fidelity/compliance, penetration, and sustainability. Additionally, 19 determinants of implementation were selected according to the Consolidated Framework for Implementation Research (Damschroder et al. 2009) and the Policy Implementation Determinants Framework (Bullock 2019). This review addresses only measures of implementation used by organizations that were mandated to take action by local, state, or national governmental agencies or legislation. As a result, organizational-level mandated policies were excluded.

We used systematic review methods developed by Lewis et al. (2018) for reviews of measures. We applied the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) to assess how well the measures are designed to work in real-world settings (pragmatic characteristics such as brevity and ease of scoring interpretation) and their reported reliability and validity.

Summary information for each measure is available on the measures information pages. Publications from this review are forthcoming.


Bullock HL. 2019. Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis in: How do systems achieve their goals? the role of implementation in mental health systems improvement (PhD Thesis), McMaster University, Hamilton, Canada. https://macsphere.mcmaster.ca/handle/11375/24820

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. https://doi.org/10.1186/1748-5908-4-50

Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, Stanick C, Halko H, Powell BJ. An updated protocol for a systematic review of implementation-related measures. Systematic Reviews. 2018;7:66. https://doi.org/10.1186/s13643-018-0728-3

Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Serv Res. 2011;38(2):65-76. https://doi.org/10.1007/s10488-010-0319-7

Stanick CF, Halko HM, Nolen EA, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transational Behavioral Medicine. 2019 (20 November). https://doi.org/10.1093/tbm/ibz164

Photo by Myles Tan on Unsplash


For questions about the website or systematic review please contact:

Peg Allen
Phone 314-935-0116
Email pegallen@wustl.edu

Or another team member:

Brown School at Washington University in St. Louis:

Stephanie Mazzucca, Meagan Pilar, Callie Walsh-BaileyRoss Brownson, Ana Baumann, Maura Kepper

Brigham Young University School of Social Work:

Cole Hooley

Kaiser Permanente Washington Health Research Institute:

Cara Lewis, Kayne Mettert, Caitlin Dorsey

Drexel University Dornsife School of Public Health:

Jonathan Purtle