Document

CMS External Quality Review (EQR) Protocols

ICR 202604-0938-021 · OMB 0938-0786 · Object 169610500.

Document Viewer [pdf]

Status: A verified source copy is available locally. Derived viewer artifacts are still being generated.

Queue: Repair is queued and waiting for the next worker pass. (priority 170 · attempts 1 · next 2026-06-11 13:49:12)

Download: Source copy (pdf)

Primary: pdfSource: application/pdf

Loading document viewer…

Document Metadata

File Type	application/pdf
File Title	CMS External Quality Review (EQR) Protocols
Conversion State	pending

Extracted Text

OMB Control No. 0938-0786
Expires: TBD

OMB Control No. 0938-0786
Expires: December 31, 2025
CMS External Quality Review (EQR) Protocols
February 2023
U.S. Department of Health and Human Services logo
Centers for Medicare & Medicaid Services (CMS) logo
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required
to complete this information collection is estimated to average 7,688 hours per response for state governments and 185 hours per
response for the private sector, including the time to review instructions, search existing data resources, gather the data needed,
and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or
suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop
C4-26-05,
[Month]Baltimore,
2026 Maryland 21244-1850.

This page left blank for double-sided copying.

CMS External Quality Review (EQR) Protocols Table
of Contents

Introduction ................................................................................................................................................... 1
Background ........................................................................................................................................... 1
Managed Care Quality Activities and the EQR Process ....................................................................... 4
Federal Financial Participation for EQR ................................................................................................ 7
Overview of the EQR Protocols ............................................................................................................ 8
Considerations Before Conducting EQR-Related Activities................................................................ 10
Nonduplication for Mandatory EQR-Related Activities ....................................................................... 12
EQR Reporting ............................................................................................................................................ 15
Tips for Drafting Compliant EQR Technical Reports .......................................................................... 15
Tips for Drafting an Effective EQR Technical Report.......................................................................... 22
State Review of EQR Technical Reports ............................................................................................ 24
Posting and Submitting EQR Technical Reports ................................................................................ 25
Getting Started on the EQR Protocols ........................................................................................................ 26
Mandatory EQR-Related Activities...................................................................................................... 27
Optional EQR-Related Activities ......................................................................................................... 28
Appendices .......................................................................................................................................... 29
For Further Information ....................................................................................................................... 30
Protocol 1. Validation of Performance Improvement Projects .................................................................... 31
Background ......................................................................................................................................... 31
Getting Started on Protocol 1 .............................................................................................................. 32
Activity 1: Assess the PIP Methodology .............................................................................................. 33
Activity 2: Perform Overall Validation .................................................................................................. 44

CONTENTS | iii

Activity 3: Verify PIP Findings (Optional) ............................................................................................ 44
Activity 4: Report Results to the State................................................................................................. 46
Worksheets for Protocol 1: PIP Validation Tools and Reporting Framework ..................................... 48
Protocol 2. Validation of Performance Measures ....................................................................................... 73
Background ......................................................................................................................................... 73
Getting Started on Protocol 2 .............................................................................................................. 75
Activity 1: Perform Preliminary Review (Pre-Site Visit) ....................................................................... 77
Activity 2: Conduct MCP Site Visit ...................................................................................................... 86
Activity 3: Perform Overall Validation (Post-Site Visit) ........................................................................ 93
Activity 4: Report Results to the State................................................................................................. 94
Worksheets for Protocol 2: Performance Measure Validation Tools .................................................. 96
Protocol 3. Review of Compliance With Medicaid and CHIP Managed Care Regulations ...................... 144
Background ....................................................................................................................................... 144
Activity 1: Establish Compliance Thresholds .................................................................................... 149
Activity 2: Perform the Preliminary Review (Pre-Site Visit) ............................................................... 150
Activity 3: Conduct MCP Site Visit .................................................................................................... 152
Activity 4: Compile and Analyze Findings (Post-Site Visit) ............................................................... 156
Activity 5: Report Results to the State............................................................................................... 158
Worksheets for Protocol 3: Compliance Review Tools ..................................................................... 161
Protocol 4. Validation of Network Adequacy ............................................................................................ 242
Background ....................................................................................................................................... 242
Getting Started on Protocol 4 ............................................................................................................ 245
Activity 1: Define the Scope of the Validation of Quantitative Network Adequacy Standards .......... 248
Activity 2: Identify Data Sources for Validation ................................................................................. 251
Activity 3: Review Information Systems Underlying Network Adequacy Monitoring......................... 253
Activity 4: Validate Network Adequacy Assessment Data, Methods, and Results ........................... 255
Activity 5: Communicate Preliminary Findings to Each MCP............................................................ 257
Activity 6: Report Results to the State............................................................................................... 257
Worksheets For Protocol 4: Network Adequacy Validation Tools..................................................... 260
Protocol 5. Validation of Encounter Data Reported by the Managed Care Plan ...................................... 277
iv | CONTENTS

Background ....................................................................................................................................... 277
Getting Started on Protocol 5 ............................................................................................................ 278
Activity 1: Review State Requirements ............................................................................................. 280
Activity 2: Review the MCP’s Capability............................................................................................ 281
Activity 3: Analyze Electronic Encounter Data .................................................................................. 284
Activity 4: Review Medical Records .................................................................................................. 290
Activity 5: Report Results to the State............................................................................................... 292
Worksheets for Protocol 5: Encounter Data Tables .......................................................................... 293
Protocol 6. Administration or Validation of Quality of Care Surveys ......................................................... 304
Background ....................................................................................................................................... 304
Getting Started on Protocol 6 ............................................................................................................ 305
Section I. Administering a Survey ..................................................................................................... 308
Activity I.1: Identify the Survey Purpose, Objectives, and Audience ................................................ 308
Activity I.2: Develop a Work Plan ...................................................................................................... 309
Activity I.3: Select the Survey Instrument.......................................................................................... 309
Activity I.4: Develop the Sampling Plan ............................................................................................ 313
Activity I.5: Develop a Strategy to Maximize Response.................................................................... 315
Activity I.6: Develop a Quality Assurance Plan ................................................................................. 318
Activity I.7: Implement the Survey According to the Work Plan ........................................................ 319
Activity I.8: Prepare and Analyze Survey Data and Present Results in a Final Report .................... 320
Section II. Validating a Survey .......................................................................................................... 322
Activity II.1: Review the Survey Purpose, Objectives, and Audience ............................................... 323
Activity II.2: Review the Work Plan ................................................................................................... 323
Activity II.3: Review the Reliability and Validity of the Survey Instrument ........................................ 323
Activity II.4: Review the Sampling Plan ............................................................................................. 324
Activity II.5: Review the Adequacy of the Response Rate ................................................................ 326
Activity II.6: Review the Quality Assurance Plan .............................................................................. 326
Activity II.7: Review the Survey Implementation ............................................................................... 326
Activity II.8: Review the Survey Data Analysis and Final Report ...................................................... 327
Technical Appendix for Protocol 6: Understanding Potential Sources of Survey Error .................... 327
CONTENTS | v

Worksheets for Protocol 6: Survey Administration and Validation Tools .......................................... 330
Protocol 7. Calculation of Additional Performance Measures................................................................... 339
Background ....................................................................................................................................... 339
Getting Started on Protocol 7 ............................................................................................................ 340
Activity 1: Prepare for Measurement ................................................................................................. 341
Activity 2: Calculate Measures .......................................................................................................... 343
Activity 3: Report Results to the State............................................................................................... 347
Worksheets for Protocol 7: Performance Measure Calculation Tools .............................................. 349
Protocol 8. Implementation of Additional Performance Improvement Projects ........................................ 357
Background ....................................................................................................................................... 357
Getting Started on Protocol 8 ............................................................................................................ 358
Activity 1: Select the PIP Topic ......................................................................................................... 360
Activity 2: Define the PIP Aim Statement .......................................................................................... 361
Activity 3: Identify the PIP Population ............................................................................................... 361
Activity 4: Use Sound Sampling Methods ......................................................................................... 361
Activity 5: Select the PIP Variables ................................................................................................... 361
Activity 6: Collect Valid and Reliable Data ........................................................................................ 363
Activity 7: Analyze Data and Interpret Results .................................................................................. 363
Activity 8: Review Improvement Strategies ....................................................................................... 364
Activity 9: Assess Whether Significant and Sustained Improvement Occurred ................................ 364
Activity 10: Report Results to the State............................................................................................. 365
Protocol 9. Conducting Focus Studies of Health Care Quality ................................................................. 366
Background ....................................................................................................................................... 366
Getting Started on Protocol 9 ............................................................................................................ 367
Activity 1: Select the Study Topic(s).................................................................................................. 368
Activity 2: Define the Study Question(s)............................................................................................ 369
Activity 3: Select the Study Variable(s) ............................................................................................. 370
Activity 4: Develop a Plan to Study the Population ........................................................................... 372
Activity 5: Collect Data ...................................................................................................................... 373
Activity 6: Analyze and Interpret Study Results ................................................................................ 374
vi | CONTENTS

Activity 7: Report Results to the State............................................................................................... 375
Protocol 10. Assist with the Quality Ratings for Medicaid and CHIP Quality Rating System ................... 377
Background ....................................................................................................................................... 377
Getting Started on Protocol 10 .......................................................................................................... 380
Section I. Preparing for MAC QRS EQR Activities ........................................................................... 384
Activity I.1: Identify the Quality Measures to be Calculated .............................................................. 385
Activity I.2: Identify MAC QRS Data Sources.................................................................................... 385
Activity I.3: Confirm All Measures Calculated for Each MCP ............................................................ 386
Activity I.4: Report Findings and Next Steps to the State ................................................................. 386
Section II. Validating MAC QRS Data ....................................................................................................... 387
Activity II.1: Review MCP Data and Information Systems Capabilities ............................................. 387
Activity II.2: Validate MAC QRS Data ............................................................................................... 387
Activity II.3: Report Results to the State............................................................................................ 391
Section III. Calculate MAC QRS Measure Performance Rates ................................................................ 391
Activity III.1: Collect Data .................................................................................................................. 392
Activity III.2: Integrate Data ............................................................................................................... 392
Activity III.3: Calculate Measure Performance Rates........................................................................ 393
Activity III.4: Stratify Measure Performance Rates ........................................................................... 393
Activity III.5: Validate Measure Performance Rates (Optional) ......................................................... 396
Activity III. 6: Report Final Results to the State ................................................................................. 396
Worksheets for Protocol 10: Assist with the Quality Ratings for Medicaid and CHIP Quality
Rating System ........................................................................................................................... 398
Protocol 11. Assisting with Evaluation of Managed Care Quality Strategies, State Directed
Payments, and In Lieu of Services and Settings .............................................................................. 407
Background ....................................................................................................................................... 407
Getting Started on Protocol 11 .......................................................................................................... 409
Activity 1: Develop an Evaluation Plan.............................................................................................. 410
Activity 2: Conduct the Evaluation..................................................................................................... 428
Activity 3: Report Results to the State............................................................................................... 430
Worksheets for Protocol 11: Evaluation Support Tools .................................................................... 432
Appendix A. EQR Reporting Tools ............................................................................................................ A.1
CONTENTS | vii

Worksheets for Appendix A ................................................................................................................ A.2
Appendix B. Information Systems Capabilities Assessment ..................................................................... B.1
Background ........................................................................................................................................ B.1
Getting Started on the ISCA ............................................................................................................... B.4
Activity 1: MCP Completes the ISCA Tool ......................................................................................... B.5
Activity 2: Perform Preliminary ISCA Review ..................................................................................... B.5
Activity 3: Conduct MCP Site Visit ..................................................................................................... B.6
Activity 4: Compile and Analyze ISCA Findings ................................................................................. B.6
Activity 5: Draft ISCA Summary for EQR Technical Report ............................................................... B.7
Worksheets for Appendix B ................................................................................................................ B.9
Appendix C. Sampling Approaches for EQR Data Collection Activities ....................................................C.1
Background ........................................................................................................................................C.1
Types of Sampling Approaches .........................................................................................................C.1
Calculating Minimum Sample Sizes for EQR Data Collection Activities ............................................C.3
Documenting Sampling Methods for EQR Data Collection Activities ................................................C.5
Appendix D. Acronyms Used In the Protocols ...........................................................................................D.1
Appendix E. EQR Glossary of Terms ........................................................................................................ E.1

viii | CONTENTS

Worksheets
Worksheet 1.1. Review the Selected PIP Topic ......................................................................................... 49
Worksheet 1.2. Review the PIP Aim Statement.......................................................................................... 50
Worksheet 1.3. Review the Identified PIP Population ................................................................................ 51
Worksheet 1.4. Review the Sampling Method ............................................................................................ 52
Worksheet 1.5. Review the Selected PIP Variables and Performance Measures ..................................... 53
Worksheet 1.6. Review the PIP Data Collection Procedures ..................................................................... 56
Worksheet 1.7. Review Data Analysis and Interpretation of PIP Results ................................................... 59
Worksheet 1.8. Assess the PIP Improvement Strategies ........................................................................... 60
Worksheet 1.9. Assess the Likelihood that a PIP Resulted in Significant and Sustained
Improvement .................................................................................................................................. 62
Worksheet 1.10. Perform Overall Validation of PIP Results ....................................................................... 63
Worksheet 1.11. Framework for Summarizing Information about PIPs ...................................................... 64
Worksheet 1.12. PIP Validation Reporting.................................................................................................. 67
Example of Worksheet 1.12. PIP Validation ............................................................................................... 69
Worksheet 1.13. PIP Summary Reporting .................................................................................................. 70
Example Worksheet 1.13. PIP Summary.................................................................................................... 71
Worksheet 2.1. List of Performance Measures to be Validated.................................................................. 98
Worksheet 2.2. Performance Measure Validation Template ...................................................................... 99
Example of Worksheet 2.2. Performance Measure Validation Template ................................................. 103
Worksheet 2.3. Medical Record Review Validation Template .................................................................. 107
Worksheet 2.4. Potential Documents and Processes for Review............................................................. 111
Worksheet 2.5. Interview Guide for MCP Data Integration and Control Personnel .................................. 116
Worksheet 2.6. Data Integration and Control Findings Tool ..................................................................... 118
Worksheet 2.7. Data and Processes Used to Produce Performance Measures: Documentation
Review Checklist .......................................................................................................................... 120
Worksheet 2.8. Data and Processes Used to Produce Performance Measures: Findings ...................... 122
Worksheet 2.9. Policies, Procedures, and Data Used to Produce Performance Measures: Review
Checklist ....................................................................................................................................... 124
Worksheet 2.10. Measure Validation Findings ......................................................................................... 126

CONTENTS | ix

Worksheet 2.11. Interview Guide for Assessing Processes and Procedures Used to Produce
Numerators and Denominators .................................................................................................... 130
Worksheet 2.12. Policies, Procedures, and Data Used to Implement Sampling: Review Checklist ........ 131
Worksheet 2.13. Sampling Validation Findings ........................................................................................ 132
Worksheet 2.14. Framework for Summarizing Information About Performance Measures ..................... 134
Example of Worksheet 2.14. Framework for Summarizing Information about Performance
Measures ..................................................................................................................................... 136
Worksheet 2.15. Performance Measure Validation Reporting.................................................................. 138
Examples of Worksheet 2.15 Performance Measure Validation Reporting.............................................. 141
Worksheet 2.16. Performance Measure Reporting .................................................................................. 143
Example Worksheet 2.16. Performance Measure Reporting ................................................................... 143
Worksheet 3.1. Compliance Review ......................................................................................................... 162
Worksheet 3.2. Compliance Definitions .................................................................................................... 195
Worksheet 3.3. Sample Site Visit Agenda ................................................................................................ 197
Worksheet 3.4. Compliance Interview Questions ..................................................................................... 198
Worksheet 3.5. Timeline and Crosswalk of State and Federal Standards Reviewed Templates ............ 237
Examples of Worksheet 3.5 Timeline and Crosswalk of State and Federal Standards Reviewed
Templates .................................................................................................................................... 239
Worksheet 3.6. Compliance Review Findings Reporting Template.......................................................... 241
Examples of Worksheet 3.6 Compliance Review Reporting Template .................................................... 241
Worksheet 4.1. State Network Adequacy Standards to be Validated ...................................................... 261
Example of Worksheet 4.1. State Network Adequacy Standards to Be Validated ................................... 262
Worksheet 4.2. Network Adequacy Indicators to be Validated ................................................................. 263
Example Worksheet 4.2 State Network Adequacy Standards to Be Validated ........................................ 263
Worksheet 4.3. Data Sources for Network Adequacy Validation.............................................................. 265
Example Worksheet 4.3 Data Sources for Network Adequacy Validation ............................................... 266
Worksheet 4.4. Network Adequacy Data Concerns Identified in Review of ISCA.................................... 267
Example Network Adequacy Data Concerns Identified in Review of ISCA Worksheet............................ 267
Worksheet 4.5. Assessment of Network Adequacy Data Sources not Reviewed in the ISCA................. 268
Worksheet 4.6. Assessment of MCP Network Adequacy Data, Methods, and Results ........................... 269
Worksheet 4.7. Summary of Network Adequacy Validation Findings ...................................................... 274

x | CONTENTS

Worksheet 4.8. Recommendations to Improve MCP Assessment of Network Adequacy ....................... 276
Worksheet 5.1. Specification of Acceptable Error Rates and Identified Areas of Concern ...................... 294
Worksheet 5.2. Data Element Validity Requirements ............................................................................... 295
Example Worksheet 5.2. Data Element Validity Requirements ................................................................ 295
Worksheet 5.3. Evaluation of Submitted Fields ........................................................................................ 296
Worksheet 5.4. Benchmark Utilization Rates............................................................................................ 297
Worksheet 5.5. Medical Record Review for Encounter Data Validation................................................... 299
Worksheet 5.6. Medical Record Review Results Summary Sheet ........................................................... 301
Worksheet 5.7. Suggested Format for Reporting Encounter Data Validation Information in the
EQR Technical Report ................................................................................................................. 302
Worksheet 6.1. Survey Purpose, Objectives, and Audience .................................................................... 331
Worksheet 6.2. Survey Work Plan ............................................................................................................ 332
Worksheet 6.3. Survey Instrument ............................................................................................................ 333
Worksheet 6.4. Survey Sampling Plan ..................................................................................................... 334
Worksheet 6.5. Strategy to Maximize Survey Response.......................................................................... 335
Worksheet 6.6. Survey Quality Assurance Plan ....................................................................................... 336
Worksheet 6.7. Survey Implementation According to the Work Plan ....................................................... 337
Worksheet 6.8. Survey Data Analysis and Final Report ........................................................................... 338
Worksheet 7.1. List of Performance Measures to be Calculated.............................................................. 350
Example Worksheet 7.1 List of Performance Measure to be Calculated ................................................. 350
Worksheet 7.2. Companion Performance Measurement Tool.................................................................. 351
Worksheet 7.3. Data Element Master Checklist ....................................................................................... 353
Worksheet 7.4. Data Availability and Data Quality ................................................................................... 354
Worksheet 7.5. File Format for Transmission of Claims Data Template .................................................. 355
Example Worksheet 7.5. File Format for Transmission of Claims Data Template ................................... 355
Worksheet 10.1. Assigning MAC QRS Measures to MCPs Template ..................................................... 399
Worksheet 10.2. MAC QRS Data Needs Template .................................................................................. 400
Worksheet 10.3. MAC QRS Measure Reporting Requirements Template............................................... 401
Example of Worksheets 10.1 – 10.3 ......................................................................................................... 402
Table 1. Example State for Worksheets 10.1-10.3 ................................................................................... 402

CONTENTS | xi

Example of Worksheet 10.1. Assigning MAC QRS Measures to MCP Template .................................... 403
Example of Worksheet 10.2. MAC QRS Data Requirements Template ................................................... 404
Example of Worksheet 10.3. MAC QRS Measure Reporting Requirements Template ........................... 406
Worksheet 11.1. Evaluation Setup ............................................................................................................ 433
Worksheet 11.2. Evaluation Metric Identification Template ...................................................................... 436
Example of Worksheet 11.2. Evaluation Metric Identification ................................................................... 437
Worksheet 11.3. Evaluation Metrics, Baseline, and Performance Targets .............................................. 438
Example Worksheet 11.3. Evaluation Metrics, Baseline, and Performance Targets ............................... 439
Worksheet 11.4. Analysis Plan Template ................................................................................................. 440
Worksheet 11.5. Evaluation Timeline ....................................................................................................... 444
Example Worksheet 11.5. Evaluation Timeline ........................................................................................ 446
Worksheet 11.6. Checklist for Evaluation Report ..................................................................................... 448
Worksheet 11.7. Evaluation Performance Metric Results Template ........................................................ 450
Worksheet A.1 EQR Reporting Checklist .................................................................................................. A.2
Worksheet A.2. MCP Information .............................................................................................................. A.3
Worksheet B.1. Information System Capabilities Assessment (ISCA) Tool .............................................. B.9
Worksheet B.2. Information System Review Worksheet & Interview Guide............................................ B.34

xii | CONTENTS

Introduction
Background
Together, Medicaid and the Children’s Health Insurance Program (CHIP) cover almost 80
million people, 1 representing about 1 in 4 people in the United States, and 41 percent of all
births. 2 Nationally, 72 percent of adults and children enrolled in Medicaid and CHIP obtain their
care through managed care plans (MCPs), although the rate of managed care enrollment in states
using a managed care delivery system varies widely. 3 The federal requirements related to Medicaid
managed care quality, including the external quality review (EQR) process, were established in
statute at section 1932(c) of the Social Security Act (the Act) and are set forth in 42 C.F.R.
438(c). The same statutory federal requirements were made applicable to CHIP managed care
quality through section 2103(f)(3) of the Act and are set forth in 42 C.F.R. 457.1240 and 1250.
Box 1 defines key terms related to the EQR process.
Box 1. Key Definitions Related to the External Quality Review Process
• Managed care plan (MCP). Encompasses managed care organizations (MCOs), prepaid inpatient health plans
(PIHPs), and prepaid ambulatory health plans (PAHPs).
• External quality review (EQR). EQR is the analysis and evaluation of aggregated information on quality,
timeliness, and access to the health services that an MCP or its contractors furnish to Medicaid beneficiaries [see
42 C.F.R. 438.320]. EQR can only be conducted by a qualified EQRO.
• External quality review organization (EQRO). An EQRO is an organization that meets the competence and
independence requirements set forth in 42 C.F.R. 438.354, and performs EQR, EQR-related activities, or both.
• EQR-related activities. The activities addressed in these protocols. EQR-related activities produce the data used
by an EQRO to complete the annual EQR. EQR-related activities may be conducted by the state, its agent that is
not an MCP, or an EQRO [see 42 C.F.R. 438.358].

The Centers for Medicare & Medicaid Services (CMS) published the Medicaid and CHIP
managed care final rule in May 2016, which aligned key rules with those of other health
insurance coverage programs, modernized how states purchase managed care for beneficiaries,
and strengthened the consumer experience and key consumer protections. 4 The rule also applied
all EQR-related activities to CHIP MCPs, extended quality provisions to additional plan types,
and added two EQR-related activities: (1) validation of network adequacy, a mandatory EQR-

Estimates are for June 2024. June 2024 Medicaid and CHIP Enrollment Data Highlights. Available at
https://www.medicaid.gov/medicaid/program-information/medicaid-and-chip-enrollment-data/report-highlights.

Data on births covered by Medicaid and CHIP in 2022 is available at https://www.cdc.gov/nchs/data/nvsr/nvsr73/nvsr7302.pdf.

Data on the percentage of Medicaid beneficiaries in comprehensive managed care, by state, is available at
https://www.medicaid.gov/medicaid/quality-of-care/downloads/beneficiary-profile-2023.pdf.

More information about the 2016 Medicaid and CHIP managed care final rule is available at
https://www.gpo.gov/fdsys/pkg/FR-2016-05-06/pdf/2016-09581.pdf.

INTRODUCTION | 1

related activity, and (2) assistance with the quality ratings for Medicaid and CHIP quality rating
system, an optional EQR-related activity.
In November 2020, CMS released revisions to the 2016 final rule. 5 The 2020 rule updated the
following EQR provisions:
•

Required states to annually identify MCOs exempt from EQR on their website in the location
where EQR technical reports are posted. States must include the name(s) of the MCO(s)
exempt from EQR, including the beginning date of the current exemption period, or that no
MCOs are exempt from EQR.

•

Clarified standards for the EQR compliance review activity described at 42 C.F.R.
438.358(b)(1)(iii) by referencing the full set of Subpart B, C, and D standards that comprise
the compliance review. The rule made further modifications to the standards subject to EQR
in each of these subparts, and CMS encourages states and EQROs to familiarize themselves
with these changes.

•

Inserted several technical revisions to CHIP regulations that cross-reference Medicaid EQR
standards to align CHIP and Medicaid EQR standards, and those quality standards relevant to
EQR reporting requirements.

In May 2024, CMS released the Managed Care Access, Finance, and Quality final rule to
advance CMS’s efforts to improve access to care, quality, and health outcomes among Medicaid
and CHIP managed care enrollees. 6 It updated the following EQR provisions:
•

Eliminated the mandatory EQR requirements from primary care case management (PCCM)
entities. States have the option to continue conducting EQR-related activities for
PCCM entities.

•

Defined the 12-month review period for mandatory EQR-related activities and clarified that
the EQR technical report must reflect activities from those 12 months. The 12-month period
begins on the first day of the most recently concluded contract year or calendar year,
whichever comes first.

•

Removed the reference to a 12-month review period for the optional EQR activities. For
optional EQR activities, states can determine the appropriate review time period based on
their intended use of the data obtained.

•

Added a new optional EQR activity to support states in evaluating their managed care quality
strategies, state directed payments, and in lieu of services and settings.

More information about the 2020 Medicaid and CHIP managed care final rule is available at
https://www.govinfo.gov/content/pkg/FR-2020-11-13/pdf/2020-24758.pdf.

More information about the 2024 Medicaid and CHIP managed care final rule is available at
https://www.govinfo.gov/content/pkg/FR-2024-05-10/pdf/2024-08085.pdf.

2 | INTRODUCTION

•

Made it easier for states to use private accreditation reviews for EQR.

•

Required states to provide more meaningful data and information in the annual EQR
technical report, such as outcomes data and results from quantitative assessments.

•

Clarified that states must notify CMS within 14 days of posting their EQR technical report
and established requirements for how long states must keep reports posted on their website.

The timeline in Figure 1 chronicles the evolution of the scope of EQR in Medicaid and CHIP.
These updated protocols reflect changes included in the 2024 rule.
Figure 1. Evolution of EQR in Medicaid and CHIP

a Balanced Budget Act of 1997 amending section 1932(c)(1)(B) of the Social Security Act.
b Section 1139A(c)(2) of the Social Security Act, as amended by section 401(a) of CHIPRA, requires the U.S.

Department of Health & Human Services (HHS) Secretary to summarize State-specific information on the quality of
health care furnished to children under titles XIX (Medicaid) and XXI (CHIP). Section 1139A(c)(1)(B) of the Act
specifically requests information gathered from the external quality reviews of managed care organizations (MCOs)
and benchmark plans.

INTRODUCTION | 3

Managed Care Quality Activities and the EQR Process
EQR is one part of a multipronged approach to Medicaid and CHIP managed care quality
(MCQ) oversight and improvement. These interconnected oversight and quality activities are
designed to inform and reinforce one another in an iterative MCQ cycle (Figure 2). For example,
state quality strategies articulate managed care priorities and activities, which are realized in part
through MCP quality assessment and performance improvement (QAPI) programs. QAPI programs
include the performance measures MCPs will report to the state and the performance improvement
projects (PIPs) they will implement. The performance measures and PIPs are then validated during
the annual EQR, with EQR findings and quality improvement recommendations included in the
EQR technical report. Finally, EQR findings and recommendations inform updates to a state’s
quality strategy (QS) and MCPs’ QAPI programs.
The MCQ cycle works best when aligned with other
Medicaid and CHIP quality oversight and improvement
efforts, such as Child and Adult Core Set measure
reporting, state delivery and payment reform initiatives,
state directed payments (SDPs), quality rating systems
(QRS), and quality improvement projects. By thinking
holistically about these components, states can maximize
their impact on quality improvement.

Managed Care Quality Improvement
For more information on managed care
quality improvement and the MCQ cycle,
visit CMS’s Medicaid and CHIP Managed
Care Quality Improvement webpage,
available at
https://www.medicaid.gov/medicaid/qualityof-care/medicaid-managed-carequality/managed-care-qualityimprovement/index.html.

Figure 2. Relationship between State Medicaid and CHIP Managed Care Quality Initiatives

4 | INTRODUCTION

Figure 3 outlines the EQR process. States using a managed care delivery system for all or some
of their Medicaid and CHIP beneficiaries are required to contract with a qualified independent
EQRO to conduct an annual EQR to assess and monitor the quality of care provided to Medicaid
and CHIP beneficiaries enrolled in MCPs and to identify opportunities for quality improvement. 7
To simplify the narrative of these protocols, the term “EQRO” refers to the entity that conducts
the EQR-related activities that generate the information for the annual EQR. An EQRO is the
only entity that may conduct the annual EQR, that is, the analysis and evaluation of information
generated by the EQR-related activities (or via nonduplication, if applicable) regarding the
quality, timeliness, and access to the health care services that an MCP, or its contractors, furnish
to beneficiaries. States with both Medicaid and CHIP managed care programs may elect to
contract with a single EQRO to conduct EQR of both Medicaid and CHIP or contract with
different EQROs for EQR of Medicaid and CHIP. Many states utilize the same EQRO for EQR
of both Medicaid and CHIP. The end product of the EQR is an annual EQR technical report,
which summarizes findings on access and quality of care and must be drafted by said EQRO. 8
Figure 3. The EQR Process

See 42 C.F.R. 457.1250 for CHIP regulations cross-reference to the Medicaid managed care EQR requirements at 42 C.F.R.
438.356.

See 42 C.F.R. 457.1250 for CHIP regulations cross-reference to Medicaid managed care EQR requirements at 42 C.F.R.
438.364.

INTRODUCTION | 5

The EQR process also includes a series of mandatory and optional EQR-related activities
designed to provide a sound understanding of the strengths and weaknesses of Medicaid and
CHIP MCP performance related to quality, timeliness, and access to care (see Box 2). The EQRrelated activities are intended to (1) improve states’ ability to oversee and manage the MCPs they
contract with for services and (2) help MCPs improve their performance with respect to quality,
timeliness, and access to care. The 2024 final rule clarified that the data period for the mandatory
EQR-related activities should cover the 12 months of the most recently concluded contract or
calendar year, whichever is closest to the review date, and that states must conduct the EQRrelated activities within the 12 months preceding the finalization of the annual EQR
technical report. 9
Box 2. Mandatory and Optional EQR-Related Activities
Mandatory EQR-Related Activities
• Validate PIPs
• Validate performance measures
• Review compliance with Medicaid and CHIP managed care regulations
• Validate network adequacy
Optional EQR-Related Activities
• Validate encounter data reported by MCPs
• Administer or validate quality of care surveys
• Calculate additional performance measures
• Conduct additional PIPs
• Conduct focus studies of health care quality
• Assist with quality ratings for Medicaid and CHIP Quality Rating System
• Assist with evaluations of managed care QSs, SDPs, and in lieu of services and settings

Effective implementation of EQR-related activities facilitates state efforts to purchase high-value
care (rather than volume) and to achieve higher-performing health care delivery systems for their
Medicaid and CHIP beneficiaries. States have flexibility regarding who will conduct the EQRrelated activities; they may be conducted by the state, its agent that is not an MCP, or an EQRO.
If the state elects to contract with an EQRO to conduct the EQR-related activities, this can be the
same EQRO that conducts the EQR for the state or one or more additional EQROs. 10 An
accrediting body may not serve as an EQRO for an MCP it accredited within the previous three
years. Regardless of which entity performs EQR-related activities, the EQRO must

9 See 42 C.F.R. 438.358(b)(1)(i), (ii), and (iv) and Table 2 for details on the 12-month data period for mandatory activities.
10

States may choose to contract with different entities, including more than one EQRO, for different EQR-related activities. For
example, the state might validate PIPs (Protocol 1) itself, contract with EQRO A for the validation of performance measures
(Protocol 2), and contract with EQRO B for the compliance review (Protocol 3). Said state could then contract with EQRO A,
B, or a third EQRO C to conduct the EQR and produce the EQR technical report. For information on state contracting options
for EQR, see 42 C.F.R. 438.356 (as cross referenced at 457.1250 for CHIP).

6 | INTRODUCTION

independently review and evaluate the data from all activities as part of the annual EQR and
include them in the EQR technical report.
Medicaid and CHIP MCOs, PIHPs, and PAHPs are subject to all four mandatory EQR-related
activities. Conducting EQR on PCCM entities, however, is at the state’s discretion. Table 1
shares information about which EQR-related activities are required and optional for each
MCP type.
Table 1. Application of Mandatory and Optional EQR-Related Activities by MCP Type
MCP Type
EQR-Related Activity

MCO

PIHP

PAHP

PCCM Entity

Validation of PIPs

Required

State Discretion

Validation of Performance Measures

Required

State Discretion

Review of Compliance with Medicaid
and CHIP Managed Care Regulations

Required

State Discretion

Validation of Network Adequacy

Required

State Discretion

Validation of Encounter Data Reported
by the MCP

State Discretion

Administration or Validation of Quality
of Care Surveys

State Discretion

Calculation of Additional Performance
Measures

State Discretion

Implementation of Additional PIPs

State Discretion

Conducting Focus Studies of Health
Care Quality

State Discretion

Assist with Quality Ratings for Medicaid
and CHIP QRS

State Discretion

N/A

Assist with Evaluations of Managed
Care QSs, SDPs, and in Lieu of
Services and Settings

State Discretion

States that elect to validate PCCM entity PIPs and network adequacy may not use Federal financial participation for these
activities. See the next section for more details.

Federal Financial Participation for EQR
For Medicaid programs, EQR (including the production of the EQR technical report) and EQRrelated activities performed on MCOs, as well as the production of the EQR technical report are
eligible for Federal financial participation (FFP) at a 75 percent match rate (1) when conducted
by a qualified EQRO, (2) when the EQR-related activities are completed using methodologies
consistent with the protocols contained within this document, and (3) when the state receives

INTRODUCTION | 7

approval of its EQRO contract from CMS. 11, 12 The EQRO’s analysis is eligible for the 75
percent match rate when the information from a Medicare or private accreditation review of an
MCO is used for the mandatory EQR-related activities. However, the accreditation activities that
produce the information cannot receive the match.
Medicaid programs are eligible for the 50 percent match rate if an agency other than a qualified
EQRO conducted the EQR-related activities. 13 EQR (including the production of the EQR
technical report) and EQR-related activities conducted on PIHPs, PAHPs and PCCM entities are
eligible for the 50 percent match rate. 14 For CHIP, EQR and EQR-related activities are subject to
the 10 percent administrative cap as required by section 2105(c)(2)(A) of the Act; a state is
eligible to receive the state’s enhanced CHIP FFP match rate for these activities, regardless of
which entity completes the activity.

Overview of the EQR Protocols
The EQR Protocols provide standardized, evidence-based methods for conducting each EQR
activity and provide guidance on reporting findings. Box 3 shows the general content of each
EQR protocol.
Box 3. Content of the EQR Protocols
• Purpose of the EQR-related activity.
• How to conduct the activity, including:
○ Data sources and data collection activities to promote data accuracy, validity, and reliability.
○ Proposed method(s) for analyzing and interpreting the data.
○ Instructions, guidelines, worksheets, and/or tools that may be used in implementing the protocol.

Figure 4 (next page) identifies the EQR protocols linked to each of the mandatory and optional
EQR-related activities, as well as the source of the regulations that guide the protocols. In
addition, an Information Systems Capabilities Assessment (ISCA) is a mandatory component of
the EQR as part of Protocols 1, 2, 3, and 4, as well as Protocols 5, 7, and 10 (if applicable). It

See 42 C.F.R. 433.15 and 438.370(a) and the July 10, 2016 CMCS Informational Bulletin (CIB), Federal Financial
Participation for Managed Care External Quality Review, available at https://www.medicaid.gov/federal-policyguidance/downloads/cib061016.pdf.

If the state or the state’s agent that is not an MCP conducts the EQR-related activity on an MCO, it would be eligible for the
50 percent match rate. See 42 C.F.R. 438.370(a)–(b). When information from a Medicare or private accreditation review of an
MCO is used to support one or more mandatory EQR-related activities in place of a Medicaid review, the EQRO’s analysis of
the MCO data as part of the EQR is eligible for FFP at the 75 percent rate. The accreditation activities that produce the
information are not eligible for the FFP.

See 42 C.F.R. 433.15 and 438.370(b).

See 42 C.F.R. 438.370(b). Note that this is a change from the previous regulation, under which the enhanced match was
available for EQR of PIHPs to the same extent as MCOs. For further explanation of the change, see discussion in the Medicaid
and CHIP Managed Care 2016 final rule at 81 C.F.R 27498, 27715-27716.

8 | INTRODUCTION

should be noted that several protocols are organized around site visits by the EQRO. If onsite
visits are not feasible, site visits may be conducted virtually to obtain the information specified in
the protocols.
Figure 4. Overview of the EQR Protocols

Notes: CHIP regulations at 42 C.F.R. 457.1250(a) cross-reference to all the Medicaid managed care EQR
requirements at 42 C.F.R. 438.320.

INTRODUCTION | 9

CMS is required to develop protocols to guide and support the annual EQR process and review
and update them every three years, as necessary. 15 The first set of protocols was issued in 2003
and updated in 2012, 2019, and 2023 (recall Figure 1). The 2023 updates incorporated regulatory
changes contained in the 2020 final rule, clarified federal requirements for the EQR process to
promote compliance, responded to state and EQRO feedback about the protocols, and included
the network adequacy validation protocol. This fourth revision of the EQR protocols which
incorporates regulatory changes contained in the 2024 final rule, provides states with new tools
to support EQR reporting, shares additional guidance for including meaningful data and
information in EQR technical reports, and adds protocols for the optional EQR QRS and
managed care evaluation activities. For a summary of updates, see page 12 of the Introduction.
The next section of this introductory chapter discusses practical considerations for states before
beginning the EQR-related activities. It provides tips to guide the drafting of effective EQR
technical reports that document performance regarding quality, timeliness, and access to care,
identify areas for improvement, and recommend interventions to improve the process and
outcomes of care. Links to the protocols and appendices are contained at the end of this chapter.
The five appendices are EQR Reporting Tools (Appendix A), Information Systems Capabilities
Assessment (Appendix B), Sampling Approaches for EQR Data Collection Activities
(Appendix C), Acronyms Used in the Protocols (Appendix D), and External Quality Review
Glossary of Terms (Appendix E).

Considerations Before Conducting EQR-Related Activities
EQR-related activities may be performed by the state, an agent of the state that is not an MCP, or
by an EQRO. 16 These protocols apply to EQR-related activities conducted by any of these
entities. While most states hire an EQRO to conduct the EQR-related activities, states may elect
to conduct the EQR-related activities themselves or to contract with an organization that is not an
EQRO or an agent that is not an MCP to perform these activities.
Regardless of which entity performs EQR-related activities, the data from all activities must be
independently reviewed and assessed by the EQRO as part of the annual EQR. For example, if a
state uses a Healthcare Effectiveness Data and Information Set (HEDIS®) Compliance Audit™
to meet the performance measure validation requirement, the EQRO does not need to duplicate
the audit. However, the EQRO must confirm that the audit meets federal EQR standards, such as
ensuring it was conducted by a certified HEDIS® Compliance Auditor using National
Committee for Quality Assurance (NCQA)-approved methods, that all relevant measures were
included, and that the findings are accurately reflected in the EQR technical report. The

See section 1932(c)(2)(A)(iii) of the Social Security Act, 42 C.F.R. 438.352, and 42 C.F.R. 438.358(c)(7).

See 42 C.F.R. 438.358(a).

10 | INTRODUCTION

independent review helps ensure that the audit results are reliable for oversight and quality
improvement purposes.
Preparing to conduct EQR-related activities involves several steps (see Box 4). Throughout all
EQR-related activities and the EQR technical report process, states must ensure that the privacy
of patient information is protected in a manner consistent with the Health Insurance Portability
and Accountability Act of 1996 (HIPAA) 17 Specifically, the 2016 final rule introduced
requirements that EQR technical reports do not disclose a patient’s identity or any Protected
Health Information (PHI). 18 Consistent with that obligation, states should ensure that their MCPs
comply with HIPAA and all other federal and state laws concerning confidentiality and
disclosure. The EQRO should ensure that its EQR-related data collection and reporting activities
meet these requirements.
Box 4. Steps to Prepare for EQR-Related Activities
1 Select an entity to conduct the EQR-related activity(ies).
○ Ensure that staff conducting EQR-related activity(ies) have the training and experience needed for the particular
activity(ies) they will be conducting.
2 Provide clear, written understanding of the parameters of the review.
○ List MCPs for review. Consider using the Worksheet A.1. EQR Reporting Checklist to ensure all applicable
MCPs undergo EQR and are included in the EQR technical report.
○ Select optional EQR-related activities (if applicable) in addition to the applicable mandatory EQR-related
activities.
○ Designate a time frame for review. The activities must be performed in the 12 months preceding the finalization
of the annual EQR technical report.
3 Review all applicable federal regulations, state regulations or standards, and MCP state contracts.
4 Confirm approach with entity and all EQR participants, including:
○ Each organization's responsibilities in collecting, reporting, and/or analyzing data.
○ Which regulations, contracts, and/or initiatives should be evaluated.
○ Which reviews will occur and tools used.
○ A timeline identifying the start and completion of each protocol.

See 42 C.F.R. 431 Subpart F and 457.1110.

See 42 C.F.R. 438.364(d).

INTRODUCTION | 11

Nonduplication for Mandatory EQR-Related Activities
Nonduplication is intended to reduce administrative burden on MCPs and states while still
ensuring relevant information is available to EQROs for the annual EQR. The expansion of
nonduplication to three of the mandatory EQR-related activities (Protocols 1, 2, and 3) 19 for all
Medicaid managed care MCOs, PIHPs, and PAHPs—not just those serving only beneficiaries
who are dually eligible for Medicare and Medicaid —provides additional flexibility to states to
reduce administrative burden. Nonduplication is an option for a state only when the Medicare or
private accreditation review standards are comparable to the EQR protocols (not vice versa). If a
state elects to use nonduplication, it must document in its managed care QS and its annual EQR
technical report the EQR-related activities for which it utilizes nonduplication, along with the
state’s rationale for its determination that the Medicare or private accreditation review standards
are comparable to those in these protocols. 20 The federal requirements related to nonduplication
of mandatory activities are described in 42 C.F.R. 438.360. Like Medicaid, CHIP MCPs may
submit information from a private accreditation review; however, with regard to CHIP,
information documenting compliance with Medicare Advantage standards is not applicable as
described in 42 C.F.R. 457.1250(a).
Nonduplication allows a state to use information from a Medicare or private accreditation review
of an MCP that occurred in the 12 months preceding the finalization of the annual EQR technical
report in place of generating that information through one or more of three mandatory EQRrelated activities (Protocols 1, 2, and 3). 21 To do so, the following conditions must be met:
•

The MCP is in compliance with the applicable Medicare Advantage or private accreditation
standards 22

•

The Medicare or private accreditation review standards are comparable to those established
through the EQR Protocols for the three mandatory EQR-related activities.

•

The MCP provides the state with all applicable reports, findings, and other results of the
Medicare or private accreditation review applicable to the specified EQR-related activities.

The state is responsible for providing the EQRO with all information from the Medicare or
private accreditation review, which is being used for nonduplication. The EQRO then assesses
the completeness of information from the accreditation review to determine the extent of
nonduplication, including confirming the comparable information fully meets the requirements
for completing the analysis and developing EQR findings and recommendations. If a state

Nonduplication is not an option for the fourth mandatory EQR-related activity of network adequacy validation (42 C.F.R.
438.358(b)(1)(iv)).

See 42 C.F.R. 438.360(c) and 438.340(b)(10).

Prior to issuance of the Medicaid and CHIP 2016 final rule, such information could only be used to provide information which
would otherwise be gathered from performing the mandatory EQR-related compliance review.

See 42 C.F.R. 422 subpart D.

12 | INTRODUCTION

chooses nonduplication, it must ensure the completion of any EQR-related activities (or
components of those activities) not addressed by the information from the Medicare or private
accreditation review. For example, if an accreditation review did not validate long-term services
or supports (LTSS) or other non-HEDIS® measures required by the state as a part of an MCP’s
QAPI program, that validation activity would need to be completed for those measures.
It is important to note that even when information from a Medicare or private accreditation
review does not completely meet the requirements of an activity, that information can still be
used toward meeting the nonduplication requirements. For example, nonduplication might satisfy
a subset of the regulatory requirements that are subjects of the compliance review. In this
example, the EQRO could use information from the nonduplication source for that subset of
requirements, and then the EQR-related activity would only need to be conducted on the
remaining requirements to fully assess compliance. Similarly, if a state requires its MCPs to
include 10 measures in QAPI and 5 are validated as a part of an accreditation review, only the
other 5 would need to be validated through the EQR-related activity. Validation information on
all 10 measures would then be provided to the EQRO for the EQR and reported in the EQR
technical report.
When information from a Medicare or private accreditation review of an MCP is used to support
one or more mandatory EQR-related activities, the EQRO’s data analysis is eligible for FFP. The
accreditation activities that produce the information are not eligible for the FFP. When using
nonduplication, the EQRO is responsible for reviewing findings from the Medicare and private
accreditation review and incorporating them into the EQR technical report. Note that the use of
nonduplication is at the discretion of the state, not its MCPs.
Nonduplication is not the same as exemption. In cases of nonduplication, the EQRO reviews and
incorporates work conducted by another entity—such as a HEDIS® audit—into the EQR but
still plays a role in validating and reporting the findings. An exemption means that the EQRO is
not required to conduct or review certain activities for an MCP because it meets specific CMSapproved criteria. Box 5 (next page) reviews the differences between nonduplication
and exemption.

INTRODUCTION | 13

Box 5. What is the Difference Between Nonduplication and Exemption?
Nonduplication is a way to provide information for the annual EQR without conducting part of, or all of, one or more
EQR-related activities by using information yielded by a comparable review process. Under nonduplication, an MCO,
PIHP, or PAHP is still subject to EQR and will be included in the annual EQR technical report. Nonduplication may be
used at the state’s discretion and consistent with documentation in the state’s managed care quality strategy.
Exemption is an option that allows a state to exempt an MCO (but not a PIHP or PAHP) from the annual EQR
process under certain circumstances. If a state exempts an MCO from EQR, the MCO will not be included in the
annual EQR technical report. The EQR technical report should note which MCOs were exempt within the EQR
technical report. Exemption may be used at the state’s discretion when the following three conditions are met:
• The MCO has both a current Medicare Advantage contract and a current Medicaid contract;
• The two contracts cover all or part of the same geographic area in the state; and
• The Medicaid contract has been in effect for at least two consecutive years before the exemption date, and during
those same two years, the MCO has been subject to EQR and met quality, timeliness, and access to health care
services standards for Medicaid beneficiaries.
If a state wants to exempt an MCO from EQR, it must obtain either of the following:
• For MCOs reviewed by Medicare, the state must obtain annually the most recent Medicare review findings from the
MCO, including all data, correspondence, information, and findings relevant to the MCO’s compliance with
Medicare standards for (1) access, quality assessment and performance improvement, health services, or
delegation of these activities, (2) all measures of the MCO’s performance, and (3) results and findings of all
performance improvement projects for Medicare enrollees.
• As part of the 2024 final rule, CMS removed the requirement that private, national accrediting organizations (PAOs)
apply for Medicare Advantage deeming authority from CMS in order for states to rely on PAO accreditation reviews
in lieu of EQR activities. For MCOs reviewed by a PAO, the state must require that the MCO provide a copy of all
findings from its most recent accreditation review if that review was used to meet certain requirements for Medicare
external review or to determine compliance with Medicare requirements. At a minimum, findings must include
accreditation review results of evaluation of compliance with individual accreditation standards, any deficiencies,
corrective action plans, and summaries of unmet accreditation requirements.
Each year, the state must identify MCOs exempt from EQR on its website in the same location where EQR technical
reports are posted. The state must include the name(s) of the exempt MCO(s), including the beginning date of the
current exemption period, or that no MCOs are exempt from EQR.
Complete requirements for exemption of MCOs are available at 42 C.F.R. 438.362.

14 | INTRODUCTION

EQR Reporting
A qualified EQRO 23 or the state may conduct the annual EQR, that is, the analysis and
evaluation of information generated by the EQR-related activities (or via nonduplication, if
applicable) regarding the quality, timeliness, and access to the health care services that an MCP,
or its contractors, furnish to enrollees. The end product of the EQR is an EQR technical report
summarizing EQR-related findings, 24 which must be drafted by an EQRO for the state. 25 This
section provides guidance on drafting compliant and effective reports and shares reminders for
reviewing, posting, and submitting reports.

Tips for Drafting Compliant EQR Technical Reports
Guidance on Report Content
EQROs should produce reports that meet all federal requirements. To promote compliance with
federal requirements, this section summarizes the requirements and includes considerations for
drafting EQR technical reports. Table 2 (next page) provides an overview of the required
elements in EQR technical reports, and Table 3 (begins page 18) identifies requirements for the
PIP validation (Protocol 1), performance measure validation (Protocol 2), review of compliance
with managed care and CHIP regulations (Protocol 3), and network adequacy validation
(Protocol 4). In addition to content requirements, EQR technical reports must meet federal
expectations related to the timing of EQR activities and the data periods used for the analysis.
Box 6 (page 21) summarizes these requirements.
States and their EQROs can use Worksheet 1.12, Worksheet 2.15, and Worksheet 3.5, as well as
the worksheets in Appendix A, to support the development of a complete and comprehensive
EQR technical report. The worksheets provide suggestions and best practices for sharing the
required findings and data for these mandatory activities in a concise, digestible format.

See 42 C.F.R. 438.354 for information about the competence and independence requirements for an EQRO.

24 See 42 C.F.R. 438.364 for required elements for EQR reporting.
25

CHIP regulations at 457.1250 cross reference to 42 C.F.R. 438.364.

INTRODUCTION | 15

Table 2. Required Elements in EQR Technical Reports and Considerations for Drafting EQR Technical Reports
Regulatory Reference
42 C.F.R. 438.360(b)

Requirement

Considerations for EQR Technical Reports

The EQR technical report must include information from any
Indicate whether the state exercised the nonduplication option and
Medicare or private accreditation review of an MCP that the state share information from the nonduplication activity, including the
used to provide information for the annual EQR.
validation findings for each MCP reviewed under nonduplication.

42 C.F.R. 438.310(b)(5) The EQR technical report includes all eligible Medicaid and CHIP Identify MCPs subject to EQR by MCP name, MCP type, managed
MCPs.
care authority, and population(s) served (i.e., Medicaid, CHIP,
Medicaid & CHIP) in an introduction, executive summary, or
appendix.
42 C.F.R. 438.364(a)(3) The EQR technical report must include an assessment of the
strengths and weaknesses of each MCP with respect to (a)
quality, (b) timeliness, and (c) access to the health care services
furnished by MCPs.

Include a chart summarizing each MCP’s strengths and weaknesses.

42 C.F.R. 438.364(a)(4) The EQR technical report must include recommendations for
improving the quality of health care services furnished by each
MCP.

Include recommendations for each MCP. Recommendations should
share the EQRO’s understanding of the weakness and suggest steps
for how the MCP—potentially in concert with the state—can best
address the issue. If the cause for the weakness is unclear or
unknown, the EQRO should suggest how the MCP and/or state can
identify the cause.

Highlight substantive findings concerning the extent to which each
MCP is furnishing high-quality, timely, and appropriate access to
health care services. Findings should focus on the specific strengths
and weaknesses the EQRO identified rather than on numerical
ratings or validation scores obtained under the EQRO’s review
methodology.

When determining recommendations, EQROs should consider
whether the suggested actions are within the authority of the MCP
(or state).
42 C.F.R. 438.364(a)(4) The EQR technical report must include recommendations for
Consider connecting EQR findings to the QS goals and objectives,
how the state can target goals and objectives in the QS, under
particularly in sections of the report that assess the state’s overall
438.340, to better support improvement in the quality, timeliness, performance of the quality, timeliness, and access to health care
and access to health care services furnished to Medicaid or
services when discussing strengths and weaknesses of an MCP or
CHIP beneficiaries.
activity or when discussing the basis of performance measures or
PIPs. Note when goals in the QS are considered in EQR activities
and which goals they are. Describe the relationship between the
state’s QS goals and the four mandatory EQR activities.

16 | INTRODUCTION

Regulatory Reference

Requirement

42 C.F.R. 438.364(a)(5) The EQR technical report must include methodologically
appropriate, comparative information about all MCPs.

Considerations for EQR Technical Reports
Aggregate findings across MCPs for each EQR activity and show
comparisons.
Provide context for the individual MCP to help the reader understand
the review's results and more readily determine whether issues are
localized or systemic.
Provide dates for important market events that occurred during the
EQR review period, such as when an MCP entered or exited the
market and how the events affected EQR-related activities, where
necessary.

42 C.F.R. 438.364(a)(6) The EQR technical report must include an assessment of the
degree to which each MCP has effectively addressed the
recommendations for quality improvement made by the EQRO
during the previous year’s EQR.

Include recommendations or findings issued by the state or EQRO in
the previous year’s EQR technical report and the assessment of
each MCP’s approach to addressing them. This is not a restatement
of a response or rebuttal to the recommendation by the MCP or
state. Document assessments with the same specificity used when
reporting on initial findings.

42 C.F.R. 438.364(d)

Ensure the EQR technical report is consistent with HIPAA (42 C.F.R.
431 Subpart F and 457.1110).

The EQR technical report must not disclose patient identity or
other protected health information.

Ensure MCPs comply with HIPAA and all other federal and state
laws concerning confidentiality and disclosure.
Ensure that EQR-related data collection and reporting activities are
consistent with HIPAA requirements.

INTRODUCTION | 17

Table 3. Requirements for the EQR Mandatory Activities and Considerations for Drafting EQR Technical Reports
Regulatory Reference

Requirement

Considerations for EQR Technical Reports

42 C.F.R. 438.364(a)(1)

The EQR technical report describes how data were aggregated
and analyzed and how conclusions were drawn about quality,
access, and timeliness of care.

For all four mandatory activities, include a description of (1) how
data were aggregated, (2) how they were analyzed, and (3) how
conclusions were drawn about the MCP’s ability to furnish
services. Ensure that the comparisons discuss quality, timeliness,
and access to healthcare services.

42 C.F.R. 438.364(a)(2)(iiv)

The EQR technical report must include the following for each of Objectives: Provide the state or EQRO’s aim for conducting the
the mandatory activities:
mandatory activity, including the general approach or methods of
validation used by the EQRO. The state may also include the
•
Objectives
objective or aim statement for each PIP to satisfy this criterion for
•
Technical methods of data collection and analysis
the PIP validation activity.
•
Description of data obtained including validated
Technical methods of data collection and analysis: Describe
performance measurement data
how the EQRO obtained data to conduct the validation activity. If
•

Conclusions drawn from the data

a collection tool is used, consider providing an example of the tool
format or questions asked in an appendix. Describe how data is
analyzed to connect the data requested to the analytical methods
that eventually support the conclusions drawn with those data and
analyses.
Description of data obtained: Based upon the collection efforts
above, describe the types of data obtained—information system
extracts, documents, answers to questions in data collection tools,
and others—to explain the nature of the data collected and
analyzed.
Conclusions drawn from the data: Provide the state or EQRO’s
methodology for drawing conclusions from the data obtained
through the mandatory EQR-related activity.

18 | INTRODUCTION

Regulatory Reference
42 C.F.R. 438.364(a)(2)(iii)

Requirement

Considerations for EQR Technical Reports

The EQR technical report must include PM data and any
outcomes data and results from quantitative assessments,
regardless of whether the data have been validated, for each of
the following mandatory activities:

Provide the validated PM data for each activity and results from
quantitative assessments. Outcomes data includes PIP variable
rates and network adequacy findings. Quantitative assessments
encompass specific measurements and outcomes from EQRO
analyses completed in addition to validation activities.

•

Validation of PIPs

•

Validation of Performance Measures

•

Validation of Network Adequacy

Quantitative assessments beyond reporting compliance review
findings are not required. However, states may choose to
include a quantitative summary of MCP compliance with state
and federal requirements. See Box 7 for examples of
quantitative assessments by mandatory EQR activity.

If these data are available in another publicly available document,
then the state’s EQR technical report may link to the secondary
document instead of reproducing the data. In such cases, the
EQR technical report must (1) clearly cite the source document,
(2) clearly identify the applicable reporting period, and (3)
summarize the EQRO’s validation findings, including any
concerns, methodological issues, and required corrective actions.
Any referenced links must remain publicly accessible for the
duration of the required five-year posting period for EQR technical
reports.

42 C.F.R. 438.358(b)(1)(i)

The EQR technical report must include information on the
validation of PIPs underway during the preceding 12 months.

Provide validation findings for all PIPs underway during the most
recently completed 12-month contract year or calendar year
(whichever is closer) preceding the EQR review, regardless of the
PIP implementation phase. States often link the time frame under
review to a corresponding measurement or performance period,
such as state or federal fiscal year or calendar year. If the MCP
only recently entered the market and had not begun a PIP, provide
a statement to this effect. If a multi-year PIP has not reached a
phase where the EQRO can produce a PIP validation rating,
provide a statement to this effect.

42 C.F.R. 438.358(b)(1)(ii)

The EQR technical report must include information on the
validation of each MCP’s performance measures calculated by
the state during the preceding 12 months.

Provide validation findings for all QAPI performance measures in
use during the most recently completed 12-month contract year or
calendar year (whichever is closer) preceding the EQR review,
regardless of the phase of the performance measure’s
implementation. If the MCP only recently entered the market and
therefore had not begun reporting on performance measures,
provide a statement to this effect.

INTRODUCTION | 19

Regulatory Reference
42 C.F.R. 438.358(b)(1)(iii)

Requirement

Considerations for EQR Technical Reports

The EQR technical report must include information on a review,
conducted within the previous rolling three-year period, to
determine each MCP’s compliance with the standards set forth
in 42 C.F.R. 438, part 56, 100, 114, Subpart D, and QAPI
requirements described in 42 C.F.R. 438.330.

For each federal standard, ensure that the method of compliance
review links the EQRO’s activities to the standard under review.
Further, ensure that a clear compliance determination is made and
recorded for each standard for each plan. A best practice is to list
a compliance score of a numerical or semi-quantitative nature.

The technical report must provide MCP results for the following
14 federal quality standards:

EQROs that assess domains, standards, and requirements that do
not neatly overlap with the regulatory standards should provide a
clear crosswalk of their activities to the standards under review. As
a best practice, the technical report may include a table outlining
the timeline for reviewing all standards for MCPs across the threeyear review period.

•

42 C.F.R. 438.58, 457.1212, Disenrollment

•

42 C.F.R. 438.100, 457.1220, Enrollee rights

•

42 C.F.R. 438.114, 457.1228, Emergency and poststabilization services

•

42 C.F.R. 438.206, 457.1230(a), Availability of services

•

42 C.F.R. 438.207, 457.1230(b), Assurances of adequate
capacity and services

•

42 C.F.R. 438.208, 457.1230(c) Coordination and continuity
of care

•

42 C.F.R. 438.210, 457.1230(d), Coverage and
authorization of services

•

42 C.F.R. 438.214, 457.1233(a), Provider selection

•

42 C.F.R. 438.224, 457.1233(e), Confidentiality

•

42 C.F.R. 438.228, 457.1260, Grievance and appeals
system

•

42 C.F.R. 230, 457.1233(b), Subcontractual relationships
and delegation

•

42 C.F.R. 438.236, 457.1233(c), Practice guidelines

•

42 C.F.R. 438.242, 457.1233(d), Health information system

•

42 C.F.R. 438.330, 457.1240(b), QAPI

42 C.F.R. 438.358(b)(1)(iv) The EQR technical report includes information on the validation
of MCP network adequacy during the preceding 12 months to
comply with requirements set forth in 42 C.F.R. 438.68 and, if
the state enrolls American Indians or Alaska Natives in
managed care, 42 C.F.R. 438.14(b)(1).

20 | INTRODUCTION

If the MCP only recently entered the market and is within the initial
three-year compliance review period, provide a statement to this
effect.

Provide validation findings for network adequacy during the most
recently completed 12-month contract year or calendar year
(whichever is closer) preceding the EQR review. If the MCP
recently entered the market and therefore had not begun reporting
network adequacy results, provide a statement to this effect.

Box 6. EQR Timing and Data Period Requirements
Federal EQR regulations establish two key timing requirements: (1) the period during which mandatory EQR activities
must be conducted, and (2) the data period those activities must assess. Together, these requirements determine
when EQR activities may occur and which data must be included in the review.
• Period for conducting EQR activities: Mandatory EQR activities must be conducted within the 12 months
preceding the posting of the EQR technical report (42 C.F.R. § 438.358(b)(1)). For example, if a state posts its
EQR technical report on April 30th, 2026, the activities included in that report must have been conducted between
May 1st, 2025, and April 29th, 2026.
• Data period subject to review: The data included in the review must reflect the most recently completed contract
year or calendar year closest to the start of the EQR activity (42 C.F.R. § 438.358(b)(3)). For example, if a state’s
managed care plan (MCP) contracts operate on a state fiscal year basis (July 1st–June 30th) and an EQR activity
begins on May 1st, 2025, the appropriate data period would be calendar year (CY) 2024 (the most recently
completed year at the time the activity begins). If the same activity instead begins on August 1st, 2025—after the
close of the 2024–2025 contract year—the appropriate data period would be July 1st, 2024, through June 30th,
2025 (the most recently completed contract year).
Examples Illustrating EQR Timing and Data Period Requirements

As part of the 2024 final rule, CMS clarified that EQR technical reports should include outcomes
data and results from quantitative assessments for the mandatory EQR activities. Outcomes data
include performance measure rates, PIP variable rates, and network adequacy findings.
Quantitative assessments encompass specific measurements and outcomes from EQRO analyses
completed in addition to validation activities. Box 7 (next page) shares more information on
INTRODUCTION | 21

quantitative assessments in EQR technical reports. In addition to these quantitative outcomes
data, EQR technical reports may include qualitative information to help illustrate processes and
findings.
Box 7. Quantitative Assessments Included in EQR Technical Reports
Quantitative assessments are a critical component of EQR technical reports. These assessments provide actionable
insights on MCP performance and include key findings from validation and oversight activities.
Examples of quantitative assessments include:
• Comparative Data: Comparisons to state averages, benchmarks, or percentiles to contextualize MCP performance
• Stratified Data: Metrics stratified by race, ethnicity, geography, dual eligibility status, and other relevant
demographic categories, to identify health disparities.
• Outlier Identification: Identification of potential outliers, such as data points below the 25th percentile or above the
95th percentile, which may signal areas requiring further investigation or improvement.
Specific examples by activity include:
• PIP and Performance Measure Validation: Assessment of MCP compliance with the various stages of validation,
highlighting strengths and areas for correction.
• Network Adequacy Validation: Identification of provider enrollment errors or deficiencies that MCPs must address
to meet state requirements.
• Other Validation Activities: Analysis of MCP adherence to state-established quality standards, highlighting any
gaps or inconsistencies that require resolution.
• Compliance Review Activities: Quantitative summary of MCP compliance with state and federal requirements,
including the percentage of elements fully, partially, or noncompliant; identification of repeat findings; and
documentation of corrective action requirements or remediation timelines.
These assessments not only support states in monitoring MCP compliance but also help drive quality improvement
efforts by identifying opportunities for enhanced oversight and targeted technical assistance.

Tips for Drafting an Effective EQR Technical Report
To be of greatest use to states and their quality improvement partners, EQROs should draft clear
and concise reports that highlight substantive findings and actionable recommendations. CMS
understands that states vary in the number of Medicaid and CHIP programs and MCPs. The
EQRO should prepare an aggregate report summarizing results across all MCPs and providing
state-level performance improvement recommendations. The EQR technical reports must also
meet the MCP-level reporting requirements for items such as identifying strengths and
weaknesses, assessing MCP actions to address the previous year’s recommendations, and PIP
and performance measure validation findings and outcomes data, among others. If appropriate,
the EQRO can address these MCP-level reporting requirements via tables or appendices to the
aggregate report or prepare separate reports by MCP or MCP type. For example, the EQRO may
develop one aggregate report on a state’s medical MCOs and a separate aggregate report on all of
the state’s behavioral health PIHPs. Box 8 (next page) provides considerations for using
aggregate and MCP-level reports.

22 | INTRODUCTION

Box 8. Using Aggregate and MCP-level Reports
States with large and varied managed care programs may group MCPs by type, geography, or populations served
and provide both MCP-level and aggregate reports. MCP-level reports may be effective for communicating planspecific information while aggregate reports can communicate required comparative information and analyses.
Creating multiple reports may help CMS and the public find and review information quickly, but states and EQROs
should describe the structure of reports in a statewide executive summary or in the introduction to the
aggregate report.

In addition to focusing on compliance with federal requirements, EQROs should consider these
tips for drafting an actionable, clear, and concise report.
•

Aim for clarity and concise presentation. EQR gathers and processes a substantial amount
of material. Avoid non-essential narratives to help readers identify the most relevant
information. Because not all readers have deep experience in the areas covered by EQR,
avoid technical language and jargon when possible. To maximize the interpretability of
results, provide context for all statistics included in the report.

•

Include a clickable or hyperlinked table of contents. For easy navigation throughout the
report(s), include a clickable or hyperlinked table of contents.

•

Produce a searchable PDF. To enable stakeholders to review topics of interest and facilitate
the use of the reports for topic-specific analyses, produce a searchable PDF.

•

Include a summary of the managed care programs, MCPs, and populations. To
contextualize findings, summarize the state’s managed care program, including populations
and services covered by MCPs. Programmatic and operational information could be shared in
a table with a column dedicated to each MCP, program, population, or service category.

•

Include an executive summary that highlights key findings. To help readers quickly
understand both the strengths and areas for improvement, begin the report with an executive
summary that succinctly synthesizes important program-level or statewide findings across
EQR activities. This high-level overview can help the state and its quality improvement
partners identify patterns and prioritize action areas.

•

Use MCP names when referring to plan performance. Findings and comparisons should
refer to MCPs by name to facilitate transparency and stakeholder understanding of specific
MCP performance

•

Consider displaying previous recommendations, MCP responses and actions, and new
recommendations in one chart. To enable a comprehensive view of each MCP’s
performance based on the EQR process, provide the previous EQR’s recommendations and
current EQR’s findings together.

•

Use charts to facilitate easy comparisons across MCPs. Share comparative information via
tables presenting performance measure data and, where relevant, PIP or NAV data. Provide
MCP-level data in tables and group the data by program or MCP type. For the compliance
INTRODUCTION | 23

review activities, tables or charts can display each MCP’s compliance and non-compliance
with each of the reviewed state and federal standards.

Making Strong Recommendations
Clear, actionable recommendations strengthen the EQR technical report and support continuous
quality improvement. EQRO recommendations should be grounded in validation findings as well
as other EQR assessment results and aligned with the scope of the review activity.
In addition to MCP-specific recommendations, the EQRO must provide state-level
recommendations. At a minimum, state-level recommendations must address the effectiveness
and implementation of the state’s managed care quality strategy (QS). When findings identify
broader systemic, policy, or oversight concerns, state-level recommendations may extend beyond
the QS to promote statewide improvement.
When drafting recommendations, the EQRO should:
•

Tie recommendations to findings. Recommendations should directly address
methodological concerns, implementation challenges, or opportunities for improvement
identified through the validation or review process.

•

Be specific and actionable. Recommendations should clearly describe the identified issue,
the expected improvement, and how progress can be monitored (e.g., through defined
milestones or targets). Avoid vague or general statements (e.g., “continue to improve
performance”) unless accompanied by details on the specific performance area, expected
improvement, and measurement approach.

•

Prioritize impact. When multiple findings are identified, recommendations should focus on
those most likely to improve quality, strengthen compliance, or enhance outcomes for
beneficiaries.

•

Clarify the level and locus of action. When applicable, recommendations should indicate
whether the recommendation can be addressed by the MCP independently or whether it
requires coordination with the state, providers, or other stakeholders. Recommendations
should also distinguish between those that warrant formal state follow-up and those that are
advisory in nature.

•

Ensure feasibility. Recommendations should reflect the scope of the identified issue and the
operational context of the MCP or state, fall within the authority of the Medicaid program or
its MCPs to address, and promote meaningful improvement without imposing unnecessary
burden.

Recommendations are not required in the absence of findings. If an activity meets validation or
review standards and shows no evidence that quality improvement is needed, the EQRO may
state that no follow-up actions are recommended.

24 | INTRODUCTION

State Review of EQR Technical Reports
The EQR technical report must be independently developed and produced by the EQRO. The
state may not substantively revise the EQR technical report submitted by the EQRO without
evidence that errors or key omissions occurred. However, the state is ultimately responsible for
submitting a complete report per 42 C.F.R. 438.364.
Upon receipt of EQR technical reports from the EQRO, the state should review the report for
completeness and adherence to these protocols. The state should also confirm that the technical
report includes all of the required elements set forth in 42 C.F.R. 438 Subpart E and includes a
review of all standards and regulations in Subpart E of Part 438 and other regulations
incorporated by reference. The worksheets included in Appendix A can support the state in
this review.

Posting and Submitting EQR Technical Reports
As required under 42 C.F.R. 438.10(c)(3) and 438.364(c)(2)(i), the state must post its finalized
annual technical report(s) on its website by April 30th of each year and notify CMS within 14
calendar days of the posting by submitting the link to [email protected].
All reports containing the required information to support compliance with EQR requirements
must be submitted and posted by the deadline. States must maintain at least the previous 5 years
of EQR technical reports on the state’s website as required under 42 C.F.R. 438.364(c)(2)(iii).

INTRODUCTION | 25

Getting Started on the EQR Protocols
The protocols in this document are designed to support the
completion of the EQR-related activities, which in turn help
the state meet the requirement to conduct an annual EQR of
Acronyms and Glossary
its MCPs and help contracted EQROs meet the requirements
See Appendix D for a list of
of producing an annual EQR technical report. If a state prefers acronyms used in the EQR
to use methods consistent with but not identical to these
Protocols. See Appendix E for a
glossary of terms.
protocols to conduct EQR-related activities, the state is
encouraged to discuss the alternative methods with CMS
before implementation to ensure the methods meet regulatory standards. These protocols replace
the 2023 EQR Protocols.
Use the “Go Now!” buttons to navigate to the individual protocols and the appendices.

If you have any questions related to the EQR protocols or would like to discuss alternative
methods, please contact CMS via the TA mailbox at [email protected].

26 | INTRODUCTION

Mandatory EQR-Related Activities
Protocol 1 – Validation of Performance Improvement Projects

Page 33

As part of their QAPI programs, MCPs are required to implement PIPs that focus on both
clinical and non-clinical aspects of care (42 C.F.R. 438.358(b)(i)). Protocol 1 specifies
procedures for EQROs to:
•

Assess and provide a validation score for the PIP’s methodology and the validity and
reliability of the PIP data.

•

Assess and provide a validation score of the PIP’s improvement strategies.

Protocol 2 – Validation of Performance Measures

Page 74

As part of their QAPI programs, MCPs must report standard performance measures specified by
the state. The state must provide the performance measures that must be calculated to the EQRO
and MCP, along with the specifications for the measures and the state’s reporting requirements.
Protocol 2 guides the EQRO on how to:
•

Evaluate the accuracy of the MCP-reported performance measures based on the measure
specifications and state reporting requirements.

•

Evaluate if the MCP followed the rules outlined by the state program for calculating the
measures (42 C.F.R. 438.358(b)(ii)).

This protocol also applies when a state requires its MCPs to submit data to the state so that the
state can calculate the standard performance measures.
Protocol 3 – Review of Compliance with Medicaid and CHIP
Managed Care Regulations

Page 145

The EQR is required to include a compliance review of each MCP once in a rolling 3-year
period. Protocol 3 specifies procedures to determine the extent to which MCPs comply with
standards set forth at 42 C.F.R. 438.358(b)(iii), state standards, and MCP contract requirements.
Note that the state may meet the three-year requirement in different ways: for example, it may
review all MCPs at the same time once every three years, or it may conduct a complete
compliance review on a subset of its MCPs each year during a three-year cycle.

INTRODUCTION | 27

Protocol 4 – Validation of Network Adequacy

Page 246

The state must ensure that its MCPs maintain provider networks that are sufficient to provide
timely and accessible care to its Medicaid and CHIP beneficiaries across the continuum of
services. As set forth in 42 C.F.R. 438.68, states are required to set quantitative network
adequacy standards for MCPs that account for regional factors and the needs of the state’s
Medicaid and CHIP populations.
Protocol 4 guides the EQRO in conducting the validation of network adequacy during the
preceding 12-month contract year or calendar year, whichever is closer to the review date, to
comply with requirements set forth in 42 C.F.R. 438.68 and, if the state enrolls American Indians
and Alaska Natives in the MCP, 42 C.F.R. 438.14(b)(1). This includes validating data to
determine whether the network standards, as defined by the state, were met.

Optional EQR-Related Activities
Protocol 5 – Validation of Encounter Data Reported by the MCP

Page 281

A managed care encounter is a distinct service provided to an enrollee. The state can use
encounter data to better understand the health services delivered by MCPs, assess and review
quality, monitor program integrity, and determine capitation payment rates. Protocol 5 specifies
procedures for assessing the completeness and accuracy of encounter data submitted by MCPs to
the state. It also assists in improving processes associated with collecting and submitting
encounter data from MCPs to the state.
Protocol 6 – Administration or Validation of Quality of Care Surveys Page 308

Surveys are a common method of measuring health care quality, especially consumer experience
with care. Protocol 6 specifies procedures for conducting various types of surveys and validating
those surveys.
Protocol 7 – Calculation of Additional Performance Measures

Page 343

The state can use performance measures to monitor MCPs’ performance over time, understand
their impact on the Medicaid and CHIP populations, compare MCP performance, and inform the
selection and evaluation of quality improvement activities. Protocol 7 specifies procedures for
calculating MCP performance measures in accordance with the state’s specifications. It also
supplies information to the state on the extent to which the MCP’s information system (IS)
provides accurate and complete information necessary to calculate performance measures.

28 | INTRODUCTION

Protocol 8 – Implementation of Additional Performance
Improvement Projects

Page 361

The state may conduct—or request an EQRO conduct—a PIP in addition to those MCPs are
required to conduct as part of their QAPI programs. Protocol 8 specifies procedures for
implementing additional PIPs.
Protocol 9 – Conducting Focus Studies of Health Care Quality

Page 370

The state may choose to conduct a study on a particular aspect of clinical and/or non-clinical
services provided by its MCPs. Protocol 9 specifies procedures for planning and carrying out a
focus study.
Protocol 10 – Assist with the Quality Rating for
Medicaid and CHIP Quality Rating System

Page 381

The state may choose to have its EQRO assist with the quality ratings for Medicaid and CHIP
QRS. Protocol 10 specifies procedures for validating data and calculating performance rates for
the MAC QRS.
Protocol 11 – Assist with Evaluation of Managed Care Quality Strategies,
State Directed Payments, and In Lieu of Services and Settings
Page 411

The state may engage its EQRO to assist with evaluating its managed care QSs, SDPs, and in
lieu of services and settings (ILOSs). Protocol 11 outlines procedures for assessing the
effectiveness of these approaches in advancing quality and access in Medicaid and CHIP
managed care programs.

Appendices
The EQR Protocols include five appendices supplementing the information contained in the
protocols. Use the “Go Now!” buttons to navigate to the appendices.
Appendix A. EQR Reporting Tools

Page A.1

This appendix provides tools to help states and EQROs create comprehensive, high-quality,
compliant EQR technical reports.

INTRODUCTION | 29

Appendix B. Information Systems Capabilities Assessment

Page B.1

Protocols 1, 2, 3, and 4 require the state to assess their MCPs’ IS capabilities. The state may also
perform this activity for Protocols 5, 7, and 10.The regulations at 42 C.F.R. 438.242 and
457.1233(d) also require the state to ensure that each MCP maintains a health IS that collects,
analyzes, integrates, and reports data for areas including, but not limited to, utilization,
grievances and appeals, and disenrollment for reasons other than the loss of Medicaid and CHIP
eligibility. Portions of the ISCA are voluntary; however, some components relate directly to the
mandatory EQR-related activity protocols. This appendix defines the recommended capabilities
of an MCP’s IS to meet the above-noted regulatory requirements, as well as how to assess the
strength of the MCP’s IS capabilities. It includes an overview of the processes for collecting,
processing, and reporting data and guidance for:
•

Completing the ISCA assessment (by MCPs)

•

Reviewing ISCA and accompanying documents

•

Interviewing MCP staff

•

Analyzing ISCA findings

Appendix C. Sampling Approaches for EQR Data Collection
Activities

Page C.1

This appendix provides an overview of sampling approaches that can be used in Protocols 1, 2,
5, 6, 7, 8, and 9.
Appendix D. Acronyms Used in the Protocols

Page D.1

This appendix defines acronyms used in the protocols.
Appendix E. EQR Glossary of Terms

Page E.1

This appendix defines the terms used in the protocols.

For Further Information
Technical assistance resources related to EQR, including the EQR protocols, are available on
Medicaid.gov at https://www.medicaid.gov/medicaid/quality-of-care/medicaid-managedcare-quality/quality-of-care-external-quality-review.
Please submit any questions or requests for technical assistance related to EQR to
[email protected].

30 | INTRODUCTION

Protocol 1. Validation of Performance
Improvement Projects
A Mandatory EQR-Related Activity

ACTIVITY 2: PERFORM OVERALL VALIDATION AND REPORTING OF
PIP RESULTS
ACTIVITY 3: VERIFY PIP FINDINGS (OPTIONAL)
ACTIVITY 4: REPORT RESULTS TO THE STATE

Background
States must require their Medicaid and Children’s Health Insurance Program
(CHIP) managed care plans (MCPs) to conduct performance improvement
projects (PIPs) that focus on both clinical and non-clinical areas each year as a
part of the plan’s quality assessment and performance improvement (QAPI)
program, per 42 C.F.R. 438.330 and 457.1240(b). 26, 27,28 See Box 1.1 for the
definition of a PIP.
Box 1.1. What is a PIP?
A PIP is a project conducted by the MCP that is designed to achieve significant
improvement, sustained over time, in health outcomes and enrollee experience. A PIP may
be designed to change behavior at a member, provider, and/or MCP/system level. The topic
should target improvement in relevant areas of clinical and non-clinical services.

This external quality review (EQR)-related activity validates the PIPs that the
MCP was required to conduct as part of its QAPI program. The PIP validation
activity should include PIPs underway in the 12 months preceding the period
the EQR activity is conducted, which begins on the first day of the most
recently concluded contract year or calendar year, whichever is nearest to the

26 Clinical PIPs focus on service outcomes and non-clinical PIPs focus on service delivery processes or

operational functions.

At a minimum, a single PIP that focuses on both clinical and non-clinical aspects of care may satisfy this
requirement. Otherwise, a state must require at least two PIPs, one clinical and one non-clinical.

28 All PIPs underway during the applicable 12-month data period subject to review must be included in the

annual EQR technical report ((42 C.F.R. § 438.364, 438.330(d)(3), 438.358(b)(1)(i)).

PROTOCOL ONE

ACTIVITY 1: ASSESS THE PIP METHODOLOGY

date the EQR activity began. The external quality review organization (EQRO) reviews the PIP
design and implementation using documents provided by the MCP, which may be supplemented
with interviews of MCP staff. It also assesses the improvement strategies and the likelihood that
they may lead to sustained improvement. The EQRO then reports to the state on its findings from
reviewing and validating the PIP(s) in the EQR technical report. As noted in the Introduction,
states have the option to use information from a Medicare or private accreditation review of an
MCP to provide information for the annual EQR instead of conducting this mandatory EQRrelated activity. 29, 30
A related protocol (Protocol 8) specifies procedures for implementing additional PIPs in
accordance with state specifications.

Getting Started on Protocol 1
To complete this protocol, the EQRO undertakes three required activities and one optional
activity to validate the PIPs for each MCP (Figure 1.1). All these activities must be completed 12
months preceding the finalization of the annual EQR technical report. 31

If the state elects to use nonduplication for this mandatory EQR-related activity (42 C.F.R. 438.360, Nonduplication of mandatory
activities with Medicare or accreditation review), then the state must ensure that all information from the Medicare or private
accreditation review is provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42 C.F.R.
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an option). Use of nonduplication
must be identified in the state’s quality strategy (see 42 C.F.R. 438.360(c) and 438.340(b)(10)). CHIP regulations at 457.1250 crossreferences to 42 C.F.R. 438.360 but does not allow for the use of Medicare review activities for nonduplication.

A state may not utilize nonduplication if Medicare has accepted only an attestation of an MCP’s quality improvement program
(QIP). In the context of this EQR-related activity, the QIP would have to undergo validation as part of a Medicare review for
nonduplication to be an option. See 42 C.F.R. 438.360(a)(2).
31 The “12 months preceding” refers to the timeframe during which the EQRO conducts the EQR activity prior to posting the
annual EQR technical report. This period is distinct from the 12-month data period subject to review, which reflects the most
recently concluded contract or calendar year being evaluated. See Box 6 the Introduction for additional information.
30

32 | PROTOCOL ONE

Figure 1.1. Protocol 1 Activities

Two supplemental resources are available to help EQROs validate PIPs, including:
•

Worksheets for Protocol 1. PIP Validation Tools and Reporting Framework, a set of
worksheets that can be used to guide and record answers for the validation of PIPs and
reporting of summary PIP information based on activities and associated steps in this
protocol

•

Appendix C. Sampling Approaches for EQR Data Collection Activities, which provides an
overview of sampling methods that could be used in this protocol

The remainder of this protocol outlines the steps associated with Activities 1 through 4.

Activity 1: Assess the PIP Methodology
The EQRO should complete the nine steps in Activity 1, listed below, and answer the questions
posed in each step.

PROTOCOL ONE | 33

Step 1: Review the Selected PIP Topic
PIP topics should target improvement in relevant areas of
clinical and non-clinical services. In this step, the EQRO
determines the appropriateness of the PIP topic(s),
including how the PIP topic was selected and input from
enrollees or providers. The Centers for Medicare &
Medicaid Services (CMS) suggests that PIP topics align
with CMS-identified priorities for quality improvement
(QI) 32 In addition, the state should review its performance
on the CMS Child and Adult Core Set measures 33 to
identify opportunities to improve performance through a
managed care PIP.

WORKSHEET 1.1
Resource for Activity 1, Step 1
Worksheet 1.1. Review the Selected PIP
Topic
• Provides a template for assessing the
appropriateness of the PIP topic,
including how the PIP topic was
selected, the consideration of the
CMS Child and Adult Core Set
measures, and input from enrollees
or providers

Step 2: Review the PIP Aim Statement
In this step, the EQRO assesses the appropriateness and
WORKSHEET 1.2
adequacy of the aim statement. The PIP aim statement
identifies the focus of the PIP and establishes the
Resource for Activity 1, Step 2
framework for data collection and analysis. The PIP aim
Worksheet 1.2. Review the PIP Aim
statement should define the improvement strategy,
Statement
population, and time period. It should be clear, concise,
• Provides a template for reviewing the
measurable, and answerable. Box 1.2 identifies
PIP Aim Statement
considerations for developing a PIP aim statement, and
Table 1.1 provides a critique of illustrative PIP aim statements.
Box 1.2. Considerations for Developing a PIP Aim Statement that is Clear, Concise, Measurable,
and Answerable
A PIP aim statement is clear, concise, measurable, and answerable if the statement specifies measurable variables
and analytics for a defined improvement strategy, population, and time period. Potential sources of information to help
form the PIP aim statement include:
• State data relevant to the topic being studied.
• MCP data relevant to the topic being studied.
• CMS Child and Adult Core Set measures.
• Enrollee focus groups or surveys.
• Relevant clinical literature on recommended care and external benchmarks.

More information about CMS priorities and initiatives is available at https://www.cms.gov/medicare/quality/meaningfulmeasures-initiative/cms-quality-strategy.

More information about the Child and Adult Core Sets is available at https://www.medicaid.gov/medicaid/quality-ofcare/performance-measurement/adult-and-child-health-care-quality-measures/index.html.

34 | PROTOCOL ONE

Table 1.1. Critique of Example PIP Aim Statements
Example PIP Aim Statements

Critique

Poor PIP
Aim
Statement

Does the MCP adequately address psychological
problems in patients recovering from myocardial
infarction?

Good PIP
Aim
Statement

Will the use of cognitive behavioral therapy in patients • Specifies the PIP intervention (cognitive
with depression and obesity improve depressive
behavioral therapy)
symptoms over a six-month period during 2028?
• Defines the population (patients with
depression and obesity) and the time period
(six-month period during 2028)

• The PIP intervention is not specified
• It is unclear how impact will be measured
• The population and time period are not
clearly defined

• Specifies the measurable impact (improve
depressive symptoms)

Step 3. Review the Identified PIP Population
In this step, the EQRO assesses whether the MCP clearly
WORKSHEET 1.3
identified the population for the PIP in relation to the PIP
aim statement (such as age, length of enrollment,
Resource for Activity 1, Step 3
diagnoses, procedures, and other characteristics).
Worksheet 1.3. Review the Identified
Depending on the nature of the PIP aim statement, PIP
PIP Population
population, and available data, the PIP may include the
• Provides a template for assessing
entire population or a sample of the population. PIPs that
whether the PIP population was
rely on existing administrative data, such as claims and
appropriately identified
encounter data, registry data, or vital records, are typically
based on the universe of the PIP population. PIPs that rely on either medical record review or the
hybrid method (which uses a combination of administrative data and medical record review)
typically include a representative sample of the identified population. If a sample was used for
the PIP, go to Step 4. If the entire population was studied, skip Step 4 and go to Step 5. If
Healthcare Effectiveness Data and Information Set (HEDIS®) measures and sampling
methodology are used, go to Step 5.

PROTOCOL ONE | 35

Step 4: Review the Sampling Method
In this step, the EQRO assesses the appropriateness of the
PIP’s sampling methods. Appropriate sampling methods
are necessary to ensure that the collection of information
produces valid and reliable results. Please refer to
Appendix C for an overview of sampling methodologies
applicable to PIPs. When HEDIS® measures are used, and
sampling is required (for example, for measures calculated
using the hybrid method), HEDIS® sampling
methodology should be used.

WORKSHEET 1.4
Resource for Activity 1, Step 4
Worksheet 1.4. Review the Sampling
Method
• Provides a template for reviewing the
suitability of the sampling method
based on the PIP aim statement and
population

Step 5: Review the Selected PIP Variables and Performance Measures
In this step, the EQRO assesses the variables selected for a
PIP. Variables in PIPs can take various forms as long as
the selected variables identify the MCP’s performance on
the PIP questions objectively and reliably and use clearly
defined indicators of performance (See Box 1.3).

WORKSHEET 1.5
Resource for Activity 1, Step 5
Worksheet 1.5. Review the Selected PIP
Variables and Performance Measures

• Provides a template for assessing the
The PIP should include the number and types of variables
appropriateness of selected PIP
that are adequate to answer the PIP question and for which
variables and performance measures
appropriate and reliable data are available to measure
performance and track improvement over time. Variables used in PIPs may be continuous,
categorical, or discrete (Table 1.2), and various measurement scales may be used to assess
performance (Table 1.3).

Box 1.3. Considerations for Selecting Variables for a PIP
A variable is a measurable characteristic, quality, trait, or attribute of a particular individual, object, or situation being
studied (see Table 1.2 for types of variables). When choosing variables, consider different types of variables and
choose the variables and measurement scales that are best suited to the available data, resources, and PIP aim
statement (see Table 1.3 for types of measure scales). CMS encourages MCPs to choose variables that reflect health
outcomes or that can be linked to health outcomes and that can be examined on at least a semi-annual basis.

36 | PROTOCOL ONE

Table 1.2. Types of Variables for PIPs
Variable Type
Continuous

Definition
Have a range of numerical values.
Note: Data collected for a continuous variable can be
recoded as a discrete variable (e.g., an enrollee’s
blood pressure is above or below a specified level).

Examples
Age, blood pressure, temperature,
height/weight, body mass index,
birthweight.

Categorical

Have a range of non-ordered, qualitative values (or
categories).

An enrollee survey question that asks
enrollees to identify the most important
among a list of incentives offered to
improve well-care visit rates.

Discrete

Have a limited number of possible categories.

An enrollee has/has not received a flu shot
in the past 12 months.

Note: Binary variables have two categories.

Table 1.3. Types of Measurement Scales for PIP Variables
Measurement
Scales

Definition

Example

Interval

The distances between numbers denote significant
The interval between an annual income of
and interpretable differences (e.g., dollars, degrees,
$40,000 and $30,000 = $10,000.
inches, pounds), and the differences are interpretable
as higher or lower.

Ordinal

Can be treated as quantitative in some circumstances An enrollee survey question that asks
and qualitative in others.
enrollees to rank their experience of care
on a scale from 1 (poor experience) to 5
(excellent experience).

Nominal

The set of categories for a qualitative variable.

Mode of transportation to work (car, bus,
subway, bicycle, walk).

Data availability should be considered when selecting variables for PIPs, as more frequent access
to data, such as on a monthly, quarterly, or semi-annual basis, supports continuous QI and Plan
Do Study Act (PDSA) efforts and can allow an MCP or state to correct or revise course more
quickly if needed. If MCPs collect monthly, quarterly, or semi-annual data, the MCP should use
a methodology to ensure comparability, such as a rolling 12-month methodology. CMS
encourages states to select PIP variables and performance measures that can be examined on at
least a semi-annual basis.
To the extent possible, CMS encourages MCPs to choose variables for PIPs that reflect
health outcomes. Performance measures are then used to measure these outcomes. For this
protocol, performance measures are used to monitor the performance of individual MCPs at a
point in time, to track MCP performance over time, to compare performance among MCPs, and
to inform the selection and evaluation of quality improvement activities. In addition, for this
protocol, “outcomes” are defined as changes in patient health, functional status, satisfaction, or

PROTOCOL ONE | 37

goal achievement that result from health care or supportive services. 34 For example, measures of
avoidable hospitalizations or emergency department visits can demonstrate the adequacy of
access to preventive and primary care and the effectiveness of care for acute and chronic
conditions. CMS recognizes that standardized performance measures addressing outcomes may
be limited because of the lag in observing changes in population health relative to the time frame
for the PIP measurement period. Moreover, health outcomes may be influenced by factors
outside of the organization’s control, such as poverty, genetics, and environmental factors. For
these reasons, PIP outcomes do not always need to be health outcomes per se but should be
linked to health outcomes.
Figure 1.2 (next page) provides guidance for selecting PIP performance measures for tracking
performance and improvement in outcomes over time. When selecting performance measures for
a PIP, the MCP should first consider existing measures because the specifications for these
measures often have been refined over time, may reflect current clinical guidance, and may have
benchmarks for assessing MCP performance. CMS encourages the use of the CMS Child and
Adult Core Sets, Core Quality Measure Collaborative, and certified community behavioral health
clinics (CCBHC) measures. 35 Additional examples of existing measures include the National
Committee for Quality Assurance’s (NCQA’s) HEDIS® or measures developed by the Agency
of Healthcare Research and Quality (AHRQ), such as the prevention quality indicators, inpatient
quality indicators, patient safety indicators, and pediatric quality indicators. 36

See 42 C.F.R. 438.320.

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityMeasures/Core-Measures.html. More information about measures for behavioral health
clinics is available at https://www.samhsa.gov/communities/certified-community-behavioral-health-clinics/guidance-andwebinars.

More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.

38 | PROTOCOL ONE

Figure 1.2. Guidance for Selecting PIP Performance Measures

When there are gaps in existing measures, the MCP may develop new measures based on current
clinical practice guidelines or health services research. The MCP should consider the
following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the MCP to calculate the measure reliably and
accurately? Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

PROTOCOL ONE | 39

Step 6: Review the PIP Data Collection Procedures
In this step, the EQRO assesses the validity and reliability
of the procedures the MCP used to collect the data that
inform the PIP measurements. Validity means that the data
measure what is intended to be measured, and reliability
means that the data produce consistent results.
To ensure the validity and reliability of the data collected as
part of the PIP, the MCP should develop a data collection
plan that specifies:
•

The data sources for the PIP

•

The data to be collected

•

How and when the data are to be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments used to collect the data

WORKSHEET 1.6
Resource for Activity 1, Step 6
Worksheet 1.6. Review the PIP Data
Collection Procedures
• Provides a template for reviewing the
appropriateness of study variables
and performance measures to track
improvement
• Includes an assessment of data
collection procedures overall and for
administrative and medical record
review

This step may involve three main kinds of data collection: administrative data sources, medical
record review, and a hybrid method. The hybrid method uses a mix of both types of data
collection. Procedures to collect data from administrative data systems will differ from
procedures for visual inspection of medical records or other primary source documents.
However, both types of data collection require assurances that data are valid and reliable. CMS
encourages states to utilize data sources from which they can collect data regularly (e.g.,
monthly, quarterly, and semi-annually).
Administrative data collection
Evaluating an administrative data collection methodology emphasizes the system that stores the
data and should focus on estimating the degree of completeness of the administrative data used
to measure performance and track improvement. See Section 2 of Worksheet 1.6 for a checklist
of administrative data assessment questions. In addition, refer to Protocol 5 for more information
on assuring the validity and reliability of encounter data.
Medical record review
Medical record review may be the only valid and reliable data source for some variables. (Note
that medical records may include other sources besides the individual patient medical record,
such as clinical tracking logs, manual registries, case management records, and the like.) If the
PIP requires medical record reviews, special attention should be given to the qualifications of the
medical record reviewers, the specificity of the guidelines for data collection, and plans for
40 | PROTOCOL ONE

ensuring inter- and intra-rater reliability. The reviewers should have a standard protocol for
reviewing records, have the knowledge to interpret the records, and have been trained to identify
and code the information in the records using consistent decision rules. See Section 3 of
Worksheet 1.6 for a medical record review assessment questions checklist.
Hybrid data collection
The hybrid method uses both administrative and medical record data. The hybrid method, when
available, should be used when administrative data or electronic health record (EHR) data are
incomplete or may be of poor quality, or the data elements for the measure are not captured in
administrative data.

Step 7: Review Data Analysis and Interpretation of PIP Results
In this step, the EQRO assesses the quality of the data
WORKSHEET 1.7
analysis and interpretation of PIP results. The review
assesses whether the appropriate techniques were used
Resource for Activity 1, Step 7
and if the analysis and interpretation was accurate. In
Worksheet 1.7. Review Data Analysis
addition, analysis and interpretation of the PIP data should
and Interpretation of PIP Results
be based on a continuous quality improvement
• Provides a template for assessing the
philosophy 37 and reflect an understanding of lessons
quality and completeness of the
learned and opportunities for improvement. Interpreting
analysis
the PIP results should involve assessing the causes of lessthan-optimal performance and collecting data to support the assessment.
Accurate data analysis is essential because the state or MCP may implement changes based on
the results. The primary source for the assessment should be analytic reports of PIP results
prepared by the MCP, including both baseline and repeat measurements of PIP outcomes. In
addition, the EQRO may assess the reasonableness of individual MCP results in relation to
existing state-level data, data from other MCPs, or industry benchmarks.
This protocol requires EQROs to assess the extent to which any change in performance is
statistically significant; however, it does not specify a level of statistical significance that must
be met. MCPs should indicate the level of statistical significance used in the analysis and which
findings were statistically significant.

Continuous Quality Improvement (CQI) refers to the ongoing study of processes to (1) improve services or outcomes, and (2)
prevent or minimize the chance of adverse outcomes. To do so, the organization identifies areas for improvement and tests
approaches.

PROTOCOL ONE | 41

Step 8: Assess the PIP Improvement Strategies
In this step, the EQRO assesses the appropriateness of
WORKSHEET 1.8
the interventions for achieving improvement. This
assessment builds on the interpretation of PIP results in
Resource for Activity 1, Step 8
Step 7. Significant, sustained improvement results from
Worksheet 1.8. Assess the Improvement
developing and implementing effective improvement
Strategies
strategies (including culturally and linguistically
• Provides a template for assessing
appropriate strategies for the target population).
whether the selected improvement
Selected strategies should be evidence-based, that is,
strategies were appropriate for
achieving improvement
there should be existing evidence (published or
unpublished) suggesting that the test of change would
be likely to lead to the desired improvement in processes or outcomes (as measured by the
variables). Using the criteria in Worksheet 1.8, the EQRO should assess whether there is
evidence that the selected interventions were appropriate to achieve the aim of the PIP.
A common approach to guide quality improvement work is the Institute for Healthcare
Improvement’s (IHI) Model for Improvement (see Box 1.4, next page). After using this model to
define the parameters for the improvement effort, the MCP may test changes on a small scale
using PDSA cycles. PDSA cycles provide a methodology to test changes on a small scale and
apply rapid-cycle learning principles to adjust intervention strategies throughout the
improvement. This approach involves a continuous cycle of measuring and analyzing
performance and requires frequent review and adjustment. Data are evaluated regularly, and
interventions are adapted based on what was learned. Interventions can then be scaled to larger
settings or populations if found effective. PIPs based on the Model for Improvement and PDSA
process are sometimes called rapid-cycle PIPs. The EQRO can use results from PDSA cycles to
inform its assessment of the appropriateness of interventions to achieve the aim of the PIP.

42 | PROTOCOL ONE

Box 1.4. Using the IHI Model for Improvement to Assess Improvement Strategies
The IHI Model for Improvement provides a framework for conducting improvement work. The Model asks three
questions:
• What is your aim, and by when do you want to accomplish the aim?
• How will you know that a change is an improvement?
• What changes can you put in place to achieve your aim?
A PDSA cycle is used to structure the testing of improvement strategies. The steps in the PDSA cycle are to:
• Plan. Plan the test or observation, including a plan for collecting data, and interpreting results
• Do. Try out the test on a small scale
• Study. Set aside time to analyze the data and assess the results
• Act. Refine the change, based on what was learned from the test. Determine how to sustain the intervention, if
successful
This information about the Model for Improvement and PDSA approach was adapted from the Institute for Healthcare
Improvement, available at https://www.ihi.org/resources/how-improve-model-improvement.
An additional source of information is the Agency for Healthcare Research and Quality’s Health Care Innovations
Exchange, available at https://www.ahrq.gov/health-literacy/improve/precautions/tool2b.html.

Step 9: Assess the Likelihood that Significant and Sustained Improvement
Occurred
In this step, the EQRO assesses the likelihood that
significant and sustained improvement occurred due to the
PIP. This assessment builds on findings from the two
previous steps. Box 1.5 suggests potential sources of
information for this assessment. The EQRO should review
the PIP methods and findings to assess whether there is
evidence of statistically significant improvement that may
be associated with the intervention implemented as part of
the PIP.

WORKSHEET 1.9
Resource for Activity 1, Step 9
Worksheet 1.9. Assess the Likelihood
that a PIP Resulted in Significant and
Sustained Improvement
• Provides a template for assessing
improvement as a result of the PIP

Box 1.5. Potential Sources of Supporting Information
When validating a PIP, EQROs should consider a variety of supporting information to assess the credibility of findings
and the impact of the intervention. The examples below illustrate common sources that can strengthen validation
ratings and provide important context for interpreting results:
• Statistical significance testing calculated on baseline and repeat indicator measurements (clarify that the
appropriate test was used, such as a t-test for small samples)
• Benchmarks for quality specified by the state Medicaid program or found in industry standards
• Interviews with MCP staff and providers about the implementation and results of the PIP intervention

PROTOCOL ONE | 43

In addition, the EQRO may supplement the quantitative assessment with information gathered
through interviews with MCP staff and/or providers about the implementation and results of the
PIP intervention. Qualitative information may inform the assessment of whether observed
changes were likely attributable to the PIP intervention, as opposed to a short-term event
unrelated to the intervention or random chance.
An important goal of a PIP is to demonstrate sustained improvement. The EQRO should assess
whether repeated measurements were conducted and, if so, whether a significant change in
performance relative to baseline measurement was observed. The repeat measurement should use
the same methodology as the baseline measurement. Any deviations in methodology (such as
sampling, data source, or variable definition) must be thoroughly documented. If the PIP is in the
early stages of implementation, and repeated measurements are not yet available, the analysis
plan should describe the methodology for subsequent measurement. The EQRO should state in
its final report which findings were found to be significant from a clinical and/or programmatic
perspective.

Activity 2: Perform Overall Validation
In this activity, the EQRO assesses the overall validity and
reliability of the PIP methods and findings to determine
whether it has confidence in the results. The EQRO will
assign two validation ratings based on the EQRO’s
assessment of whether the PIP (1) adhered to an acceptable
methodology for all phases of design and data collection,
conducted accurate data analysis and interpretation of PIP
results, and (2) produced evidence of improvement.

WORKSHEET 1.10
Resources for Activity 2
Worksheet 1.10. Perform Overall
Validation of PIP Results
• Provides a template to provide a
validation rating (high confidence,
moderate confidence, low
confidence, or no confidence)

To assign the validation ratings, the EQRO will review the
assessments conducted as part of the nine steps in Activity 1, and recorded in the Worksheets for
Protocol 1, or a similar tool. As studies always have weaknesses, the EQRO will need to assess
the relative strengths and weaknesses and the extent to which they affect the confidence in the
generalizability and usefulness of the PIP’s findings. CMS suggests using the following
validation rating to facilitate comparisons across PIPs and states: high confidence, moderate
confidence, low confidence, and no confidence.

Activity 3: Verify PIP Findings (Optional)
A state may request that the EQRO verify the data produced by the MCP to determine if the
baseline and repeated measurements are accurate. While the validation and reporting of the PIP
methodology and findings are mandatory activities, the verification of data or performance
measures used in the PIP is optional for EQROs. When conducted, verification of PIP findings
should align with established expectations described in Box 1.6 (next page).

44 | PROTOCOL ONE

Box 1.6. Expectations for Verification of PIP Findings
Activity 3 (Verification of PIP Findings) is an optional PIP Validation activity that a state may elect to have its EQRO
conduct. If implemented, the annual EQR technical report must describe the EQRO’s methodology and findings.
If the EQRO elects to validate measures, it is not required to validate all measures included in the PIP. For example,
the EQRO may validate performance or outcome measures while relying on reported process measures (e.g.,
measures assessing implementation progress, intervention uptake, or participation rates) without independently
validating them. If PIP measures are validated- whether outcome or process measures- the EQR technical report must
include (1) a description of the measures validated, (2) the data sources, (3) the criteria or thresholds used to assess
their validity and performance, and (4) the EQRO’s validation findings and conclusions.

Verification activities can provide added confidence in reported PIP results as they provide
greater evidence that the findings are accurate. However, verification is a resource-intensive
activity that may not be necessary. For example, verification may not be needed if the PIP uses
HEDIS® measures certified by a third party. Additionally, the Information System Capabilities
Assessment (ISCA) may provide assurances that the processes used to develop measures for the
PIP are valid and reliable (see Appendix B). Similarly, if the PIP relies on encounter data and the
EQRO has conducted encounter data validation, the optional EQR-related activity described in
Protocol 5, further assurances may be provided about the accuracy and completeness of the data
used in the PIP.
If a state opts to have the EQRO verify the accuracy of the baseline and repeated measurements,
the EQRO should focus on the processes through which data for the PIP were obtained,
processed, and analyzed. The verification process should begin with a thorough review of
existing resources:
•

Documentation produced by the MCP about the data, algorithms, and testing (e.g., code
reviews) related to the PIP data analysis.

•

The assessment of the MCP’s information system produced as part of the ISCA.

•

Any external validations of the accuracy and completeness of MCP encounter data (such as
the optional EQR-related activity).

•

Results of other EQR-related activities, such as performance measure validation or
compliance reviews.

•

Results of private accreditation reviews or state Medicaid and CHIP program audits.

If no current assessment of an MCP’s information system or encounter data exists, the state may
choose to contract this function to verify the PIP’s accuracy. Next, the EQRO should review
specific algorithms and results related to the PIP measures. Questions include:

PROTOCOL ONE | 45

•

Was the algorithm used to produce the PIP measures sound (that is, does the algorithm
measure what it is intended to measure, are the results consistent, and is the code well
documented)?

•

For measures calculated using administrative data: Did the MCP’s information system
capture enrollee information completely and accurately? To answer this question, the EQRO
may need to validate a sample of records to ensure the encounter data are complete.

•

For measures produced through medical record review: Did the MCP re-abstraction a small
subset (validation sample) of the reviewed records to ensure the abstraction was complete
and accurate? Data retrieval and analysis should be conducted on a small scale, with the
validation sample following the same rules as the original PIP.

If validation of a sample of records is performed, the EQRO should perform statistical
correlations between the validation sample and the original PIP data. A variety of statistical
methods can be applied to assess the degree of correlation between the PIP and validation
measures. Two recommended methods are the Pearson correlation coefficient for continuous
data (e.g., age, income) and the Kappa statistic for categorical data (e.g., sex, race). Assessing the
algorithm together with the integrity of the data will provide a strong indication of the accuracy
of the PIP’s findings.

Activity 4: Report Results to the State
In this activity, the EQRO reports its findings to the state. 38 When sharing findings, the EQRO
should include a description of the PIPs that were validated and the findings of the EQRO’s
validation review. The EQRO must include the validation results in the final EQR technical
report, as well as the performance measurement data (such as the PIP variable rates) and findings
from quantitative assessments conducted as part of the PIP validation activity. 39 Box 1.7 (next
page) shares additional guidance on these requirements. See “EQR Reporting” in the
Introduction for further guidance on producing a clear and concise report.

For the purposes of the EQR protocols and ease of explanation, we refer to EQROs as the entity conducting the EQR-related
activities.

CHIP regulations at 42 C.F.R. 457.1250 cross-reference to 42 C.F.R. 438.364(a)(2)(iii).

46 | PROTOCOL ONE

EQR technical reports must include all PIPs underway during the
12 months preceding the EQR activity. As a result, reports may
need to address a large number of PIPs. See Box 1.8 for guidance
on reporting on high-volume PIPs.

WORKSHEET 1.11
WORKSHEET 1.12
WORKSHEET 1.13

Box 1.8. Considerations for Reporting on High-Volume PIPs
When reporting on a high volume of concurrent PIPs, the EQR technical
report should prioritize sharing validation findings and progress toward
improvement while maintaining compliance with federal reporting
requirements. The following approaches may be used to streamline
reporting:
• Tier reporting by PIP phase. Tailor the level of detail to the PIP phase.
For example, planning-phase PIPs may focus on design validation
findings; implementation-phase PIPs could focus on progress, barriers,
and recommendations; and completed PIPs should include full outcomes
analysis, statistical results, and final conclusions regarding effectiveness
and sustainability. Note when data are unavailable due to PIP phase.
• Cross-reference previously reported information and focus on
updates. Background elements (e.g., rationale, baseline data, intervention
descriptions) that were fully described in a prior EQR technical report need
not be restated. Instead, reference and link to earlier reports and
summarize updates, new findings and recommendations. For ongoing
PIPs, emphasize validation determinations, key recommendations, and
progress toward improvement, rather than reproducing detailed PIP
documentation.

Resources for Activity 4
Worksheet 1.11. Framework for
Summarizing Information about PIPs
• Provides a structure for reporting
PIP-level information assessed
during the PIP validation activity
Worksheet 1.12. PIP Validation
Reporting Template
• Provides a template for reporting PIP
validation findings across multiple
MCPs
Worksheet 1.13 PIP Summary
Reporting Template
• Provides a template for reporting key
PIP information within the body of an
EQR technical report

• Use appendices for detailed PIP data. Include detailed PIP
documentation in appendices while keeping the main body of the EQR
technical report focused on validation conclusions, cross-cutting trends,
and recommendations.
• Organize PIPs strategically. Group PIPs by line of business, topic area,
or shared intervention focus to streamline presentation and reduce
duplicative narrative.
• Avoid duplicating publicly available MCP reports. If MCP-submitted
PIP reports are publicly available, the EQR technical report need not
reproduce the content. Instead, provide a link to PIP documentation and
summarize key validation findings, conclusions, and recommendations.

END OF PROTOCOL 1

PROTOCOL ONE | 47

Worksheets for Protocol 1: PIP Validation Tools and
Reporting Framework
Instructions. Use these or similar worksheets to assist in validating PIPs conducted by the MCP. These worksheets
provide templates for validating PIPs and a framework for reporting validated PIPs in the EQR technical report. This
tool includes the following worksheets crosswalked to the applicable Activity and Step:
Worksheet Name

Protocol Activity and Step

Worksheet 1.1. Review the PIP Topic

Activity 1. Step 1. Review the Selected PIP Topic

Worksheet 1.2. Review the PIP Aim Statement

Activity 1. Step. 2. Review the PIP Aim Statement

Worksheet 1.3. Review the Identified PIP Population

Activity 1. Step 3. Review the Identified PIP Population

Worksheet 1.4. Review the Sampling Method

Activity 1. Step 4. Review the Sampling Method

Worksheet 1.5. Review the Selected PIP Variables

Activity 1. Step 5. Review the Selected PIP Variables

Worksheet 1.6. Review the PIP Data Collection
Procedures

Activity 1. Step 6. Review the PIP Data Collection
Procedures

Worksheet 1.7. Review Data Analysis and Interpretation
of PIP Results

Activity 1. Step 7. Review Data Analysis and
Interpretation of PIP Results

Worksheet 1.8. Assess the PIP Improvement Strategies

Activity 1. Step 8. Assess the PIP Improvement
Strategies

Worksheet 1.9. Assess the Likelihood that a PIP
Resulted in Significant and Sustained Improvement

Activity 1. Step 9. Assess the Likelihood that Significant
and Sustained Improvement Occurred

Worksheet 1.10. Perform Overall Validation of PIP
Results

Activity 2. Perform Overall Validation and Reporting of
PIP Results

Worksheet 1.11. Framework for Summarizing Information Activity 4. Report Results to the State
about PIPs
Worksheet 1.12 PIP Validation Reporting Template

Activity 4. Report Results to the State

Worksheet 1.13 PIP Summary Reporting Template

Activity 4. Report Results to the State

48 | PROTOCOL ONE

Worksheet 1.1. Review the Selected PIP Topic
PIP Topic ______________________________________________________________________________
Instructions. Assess the appropriateness of the selected PIP topic by answering the following questions about the
MCP and PIP. Insert comments to explain “No” and “Not Applicable” responses.
Acronyms: CMS = Centers for Medicare & Medicaid Services; HHS = Department of Health and Human Services;
IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCP= Managed Care Plan; NA
= Not Applicable; PIP = Performance Improvement Project.
Question

Yes

Comments

1.1 Was the PIP topic selected through a comprehensive
analysis of MCP enrollee needs, care, and services (e.g.,
consistent with demographic characteristics and health
risks, prevalence of conditions, or the need for a specific
service by enrollees)? (If the state required the PIP topic,
please check “Not Applicable” and note in comments.)
1.2 Did selection of the PIP topic consider performance on
the CMS Child and Adult Core Set measures?
1.3 Did the selection of the PIP topic consider input from
enrollees or providers who are users of, or concerned
with, specific service areas? (If the state required the PIP
topic, please check “Not Applicable” and note in
comments.)
• To the extent feasible, input from enrollees who are
users of, or concerned with, specific service areas
should be obtained.
1.4 Did the PIP topic address care of special populations
or high priority services, such as:
• Children with special health care needs
• Adults with physical disabilities
• Children or adults with behavioral health issues
• People with intellectual and developmental disabilities
• People with Medicare and Medicaid dual eligibility who
use LTSS
• Preventive care
• Acute and chronic care
• High-volume or high-risk services
• Care received from specialized centers (e.g., burn,
transplant, cardiac surgery)
• Continuity or coordination of care from multiple
providers and over multiple episodes
• Appeals and grievances
• Access to and availability of care
1.5 Did the PIP topic align with priority areas identified by
HHS and/or CMS?
1.6 Overall assessment: In the comments section, note
any recommendations for improving the PIP topic.

PROTOCOL ONE | 49

Worksheet 1.2. Review the PIP Aim Statement
PIP Aim Statement ______________________________
Instructions. Assess the appropriateness of the selected PIP topic by answering the following questions. Insert
comments to explain “No” and “Not Applicable” responses.
Acronyms: NA = Not Applicable; PIP = Performance Improvement Project.
Question
2.1 Did the PIP aim statement
clearly specify the improvement
strategy, population, and time
period for the PIP?
2.2 Did the PIP aim statement
clearly specify the population for
the PIP?
2.3 Did the PIP aim statement
clearly specify the time period
for the PIP?
2.4 Was the PIP aim statement
concise?
2.5 Was the PIP aim statement
answerable?
2.6 Was the PIP aim statement
measurable?
2.7 Overall assessment: In the
comments section, note any
recommendations for improving
the PIP aim statement.

50 | PROTOCOL ONE

Yes

Comments

Worksheet 1.3. Review the Identified PIP Population
PIP Population _______________________________
Instructions. Assess whether the study population was clearly identified by answering the following questions. Insert
comments to explain “No” and “Not Applicable” responses.
Acronyms: MCP= Managed Care Plan; NA = Not Applicable; PIP = Performance Improvement Project.
Question

Yes

Comments

3.1 Was the project population clearly
defined in terms of the identified study
question (e.g., age, length of the study
population’s enrollment, diagnoses,
procedures, other characteristics)?
• The required length of time will vary
depending on the PIP topic and
performance measures.
3.2 Was the entire MCP population
included in the PIP?
3.3 If the entire population was included in
the PIP, did the data collection approach
capture all enrollees to whom the PIP
question applied?
• If data can be collected and analyzed
through an administrative data system,
it may be possible to study the whole
population. For more guidance on
administrative data collection, see
Worksheet 1.6.
3.4 Was a sample used? (If yes, use
Worksheet 1.4 to review sampling
methods).
• If the data will be collected manually
(such as through medical record
review), sampling may be necessary.
3.5 Overall assessment: In the comments
section, note any recommendations for
identifying the project population.

PROTOCOL ONE | 51

Worksheet 1.4. Review the Sampling Method
Instructions. Assess whether the sampling method was appropriate by answering the following questions. Insert
comments to explain “No” and “Not Applicable” responses. Refer to Appendix C for an overview of sampling
approaches for EQR data collection activities.
Acronyms: HEDIS® = Healthcare Effectiveness Data and Information Set; NA = Not Applicable; PIP = Performance
Improvement Project.
Overview of Sampling Method _________________________________________________
If HEDIS® sampling is used, check here and skip the rest of this worksheet.
Question
4.1 Did the sampling frame contain a
complete, recent, and accurate list of
the target PIP population?
• A sampling frame is the list from
which the sample is drawn. It
includes the universe of members of
the target PIP population, such as
individuals, caregivers, households,
encounters, providers, or other
population units eligible to be
included in the PIP. The
completeness, recency, and
accuracy of the sampling frame are
key to the representativeness of the
sample.
4.2 Did the sampling method consider
and specify the true or estimated
frequency of the event, the confidence
interval to be used, and the acceptable
margin of error?
4.3 Did the sample contain a sufficient
number of enrollees, taking into
account non-response?
4.4 Did the method assess the
representativeness of the sample
according to subgroups, such as those
defined by age, geographic location, or
health status?
4.5 Were valid sampling techniques
used to protect against bias? Specify
the type of sampling used in the
“comments” field.
4.6 Overall assessment: In the
comments section, note any
recommendations for improving the
sampling method.

52 | PROTOCOL ONE

Yes

Comments

Worksheet 1.5. Review the Selected PIP Variables and Performance Measures
Instructions. Assess whether the selected PIP variables were appropriate for measuring performance and tracking
improvement by answering the following questions. Insert comments to explain “No” and “Not Applicable” responses.
Recall that CMS encourages MCPs to choose variables for PIPs that reflect health outcomes. Performance measures
are then used to measure these health outcomes. When selecting variables, the MCP should consider existing
performance measures.
Acronyms: AHRQ = Agency for Healthcare Research and Quality; CCBHC = Certified Community Behavioral Health
Clinics; CMS = Centers for Medicare & Medicaid Services; HEDIS® = Healthcare Effectiveness Data and Information
Set; MCP= Managed Care Plan; NA = Not Applicable; PIP = Performance Improvement Project.
Selected PIP Variables and Performance Measures:
____________________________________________________________
Question
PIP Variables
5.1 Were the variables adequate to answer the
PIP question?
• Did the PIP use objective, clearly defined, timespecific variables (e.g., an event or status that
can be measured)?
• Were the variables available to measure
performance and track improvement over time?
(CMS encourages states to select variables
that can be examined on at least a semi-annual
basis
Performance Measures
5.2 Did the performance measure assess an
important aspect of care that will make a
difference to enrollees’ health or functional
status?
5.3 Were the performance measures appropriate
based on the availability of data and resources to
collect the data (administrative data, medical
records, or other sources)?
5.4 Were the measures based on current clinical
knowledge or health services research?
• Examples may include:
○ Recommended procedures
○ Appropriate utilization (hospital
admissions, emergency department visits)
○ Adverse incidents (such as death,
avoidable readmission)
○ Referral patterns
○ Authorization requests
○ Appropriate medication use

53 | PROTOCOL ONE

Yes

Comments

Question
5.5 Did the performance measures:
• Monitor the performance of MCPs at a point in
time?
• Track MCP performance over time?
• Compare performance among MCPs over
time?
• Inform the selection and evaluation of quality
improvement activities?
5.6 Did the MCP consider existing measures,
such as CMS Child and Adult Core Sets, Core
Quality Measure Collaborative, CCBHC
measures, HEDIS®, or AHRQ measures?
5.7 If there were gaps in existing measures, did
the MCP consider the following when developing
new measures based on current clinical practice
guidelines or health services research?
• Did the measure address accepted clinical
guidelines relevant to the PIP question?
• Did the measure address an important aspect
of care or operations that was meaningful to
MCP enrollees?
• Did available data sources allow the MCP to
calculate the measure reliably and accurately?
• Were all criteria used in the measure defined
clearly (such as time periods, characteristics of
eligible enrollees, services to be assessed, and
exclusion criteria)?
5.8 Did the measures capture changes in enrollee
satisfaction or experience of care?
• Although enrollee satisfaction/experience is an
important outcome of care in clinical areas,
improvement in satisfaction should not be the
only measured outcome of a clinical project.
Some improvement in health or functional
status should also be addressed
• For projects in non-clinical areas (such as
addressing access or availability of services),
measurement of health or functional status is
preferred
5.9 Did the measures include a strategy to ensure
inter-rater reliability (if applicable)?

54 | PROTOCOL ONE

Yes

Comments

Question

Yes

Comments

5.9 If process measures were used, is there
strong clinical evidence indicating that the
process being measured is meaningfully
associated with outcomes?
• This determination should be based on
published guidelines, including citations from
randomized clinical trials, case-control studies,
or cohort studies
• At a minimum, the PIP should be able to
demonstrate a consensus among relevant
practitioners with expertise in the defined area
who attest to the importance of a given process
5.10 Overall assessment: In the comments
section, note any recommendations for improving
the selected PIP variables and performance
measures.

PROTOCOL ONE | 55

Worksheet 1.6. Review the PIP Data Collection Procedures
Instructions. Assess whether the PIP data collection procedures were valid and reliable by answering the following
questions. This worksheet includes three sections: (1) overall data collection procedures, (2) data collection
procedures for administrative data sources, and (3) data collection procedures for medical record review. Insert
comments to explain “No” and “Not Applicable” responses.
Acronyms: EHR = Electronic Health Record; EVV = Electronic Visit Verification; LTSS = Long-Term Services and
Supports; NA = Not Applicable; PIP = Performance Improvement Project.

Section 1: Assessment of Overall Data Collection Procedures
Question
6.1 Did the PIP design specify a systematic
method for collecting valid and reliable data
representing the population in the PIP?
6.2 Did the PIP design specify the frequency of
data collection? If yes, what was the frequency
(for example, semi-annually)?
6.3 Did the PIP design clearly specify the data
sources?
• Data sources may include:
○ Encounter and claims systems
○ Medical records
○ Case management or electronic visit
verification systems
○ Tracking logs
○ Surveys
○ Provider and/or enrollee interviews
6.4 Did the PIP design clearly define the data
elements to be collected?
• Accurate measurement depends on clear and
concise definitions of data elements (including
numerical definitions and units of measure)
6.5 Did the data collection plan link to the data
analysis plan to ensure appropriate data would be
available for the PIP?
6.6 Did the data collection instruments allow for
consistent and accurate data collection over the
time periods studied?
6.7 If qualitative data collection methods were
used (such as interviews or focus groups), were
the methods well-defined and designed to collect
meaningful and useful information from
respondents?

56 | PROTOCOL ONE

Yes

Comments

Question

Yes

Comments

6.8 Overall assessment: In the comments section,
note any recommendations for improving the data
collection procedures.
Note: Include an assessment of data collection
procedures for administrative data sources and
medical record review noted below.

Section 2: Assessment of Data Collection Procedures for Administrative Data Sources
Question

Yes

Comments

6.9 If inpatient data were used, did the data
system capture all inpatient
admissions/discharges?
6.10 If primary care data were used, did primary
care providers submit encounter or utilization
data for all encounters?
6.11 If specialty care data were used, did
specialty care providers submit encounter or
utilization data for all encounters?
6.12 If ancillary data were used, did ancillary
service providers submit encounter or utilization
data for all services provided?
6.13 If LTSS data were used, were all relevant
LTSS provider services included (for example,
through encounter data, case management
systems, or EVV systems)?
6.14 If EHR data were used, were patient,
clinical, service, or quality metrics validated for
accuracy, completeness, and comparability
across systems?

Section 3: Assessment of Data Collection Procedures for Medical Record Review
Question

Yes

Comments

6.15 Was a list of data collection personnel and
their relevant qualifications provided?
• Data collection personnel require the
conceptual and organizational skills to
abstract data. These skills will vary depending
on the nature of the data and the degree of
professional judgment required. For example,
trained medical assistants or medical records
clerks may collect data if the abstraction
involves verifying the presence of a diagnostic
test report. However, experienced clinical staff
(such as registered nurses) should be used to
extract data to support a judgment about
whether clinical criteria are met

PROTOCOL ONE | 57

Question
6.16 For medical record review, were inter-rater
and intra-rater reliability described?
• The PIP should also consider and address
intra-rater reliability (i.e., reproducibility of
judgments by the same abstractor at a
different time)
6.17 For medical record review, were guidelines
for obtaining and recording the data developed?
• A glossary of terms for each project should be
developed before data collection begins to
ensure consistent interpretation among and
between data collection staff
• Data collection staff should have clear, written
instructions, including an overview of the PIP,
how to complete each section of the form or
instrument, and general guidance on handling
situations not covered by the instructions. This
is particularly important when multiple
reviewers are collecting data

58 | PROTOCOL ONE

Yes

Comments

Worksheet 1.7. Review Data Analysis and Interpretation of PIP Results
Instructions. Assess whether the data analysis and interpretation were appropriate by answering the following
questions. Insert comments to explain “No” and “Not Applicable” responses.
Acronyms: MCP= Managed Care Plan; NA = Not Applicable; PIP = Performance Improvement Project.
Question

Yes

Comments

7.1 Was the analysis conducted in accordance
with the data analysis plan?
7.2 Did the analysis include baseline and
repeat measurements of project outcomes?
7.3 Did the analysis assess the statistical
significance of any differences between the
initial and repeat measurements?
7.4 Did the analysis account for factors that
may influence the comparability of initial and
repeat measurements?
7.5 Did the analysis account for factors that
may threaten the internal or external validity of
the findings?
7.6 Did the PIP compare the results across
multiple entities, such as patient subgroups,
provider sites, or MCPs?
• Comparing the performance across multiple
entities involves greater statistical design
and analytical considerations than those
required for a project assessing the
performance of a single entity, such as an
MCP, over time.
7.7 Were PIP results and findings presented
concisely and easily understood?
7.8 To foster continuous quality improvement,
did the analysis and interpretation of the PIP
data include lessons learned about less-thanoptimal performance?
• Analysis and interpretation of the PIP data
should be based on a continuous
improvement philosophy and reflect on
lessons learned and opportunities for
improvement
7.9 Overall assessment: In the comments
section, note any recommendations for
improving the analysis and interpretation of
PIP results.

PROTOCOL ONE | 59

Worksheet 1.8. Assess the PIP Improvement Strategies
Instructions. Assess whether the selected improvement strategies were appropriate for achieving improvement by
answering the following questions. Insert comments to explain “No” and “Not Applicable” responses.
Acronyms: NA = Not Applicable; PDSA = Plan Do Study Act; PIP = Performance Improvement Project.
Question

Yes

Comments

8.1 Was the selected improvement strategy
evidence-based? Was there existing evidence
(published or unpublished) suggesting that the test of
change would be likely to lead to the desired
improvement in processes or outcomes (as
measured by the PIP variables)?
8.2 Was the strategy designed to address root
causes or barriers identified through data analysis
and quality improvement processes?
• Interventions that might have a short-term effect
but that are unlikely to generate long-term change
(such as a one-time reminder letter to enrollees or
providers) are insufficient
• It is expected that interventions associated with
significant improvement will be system
interventions (such as educational efforts, policy
changes, or targeting of additional resources)
• It is expected that interventions should be
measurable on an ongoing basis (e.g., quarterly,
monthly) to monitor intervention progress
8.3 Was the rapid-cycle PDSA approach used to test
the selected improvement strategy?
• The steps in the PDSA cycle 40 are to:
○ Plan. Plan the test or observation, including a
plan for collecting data and interpreting the
results
○ Do. Try out the test on a small scale
○ Study. Set aside time to analyze the data and
assess the results
○ Act. Refine the change based on what was
learned from the test. Determine how to
sustain the intervention, if it is successful
• If tests of change were not successful (i.e., did not
achieve significant improvement), a process to
identify possible causes and implement solutions
should be identified

Institute for Healthcare Improvement: Science of Improvement, Testing Changes. Available at https://www.ihi.org/howimprove-model-improvement-testing-changes.

60 | PROTOCOL ONE

Question

Yes

Comments

8.4 Was the strategy culturally and linguistically
appropriate? 41
8.5 Was the implementation of the strategy designed
to account or adjust for any major confounding
variables that could have a noticeable impact on PIP
outcomes (e.g., patient risk factors, Medicaid and
CHIP program changes, provider education, clinic
policies or practices)?
8.6 Building on the findings from the data analysis
and interpretation of PIP results (Step 7), did the PIP
assess the extent to which the improvement strategy
was successful and identify potential follow-up
activities?
8.7 Overall assessment: In the comments section,
note any recommendations for improving the
implementation strategies.

More information on culturally and linguistically appropriate services is available at
http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=15.

PROTOCOL ONE | 61

Worksheet 1.9. Assess the Likelihood that a PIP Resulted in Significant and Sustained
Improvement
Instructions. Assess the likelihood that significant and sustained improvement occurred by answering the following
questions. Insert comments to explain “No” and “Not Applicable” responses.
Acronyms: NA = Not Applicable; PIP = Performance Improvement Project.
Question
9.1 Was the same methodology used for baseline
and repeat measurements?
9.2 Was there any quantitative evidence of
improvement in processes or outcomes of care?
9.3 Was the reported performance improvement
likely to result from the selected intervention?
• It is not necessary to demonstrate conclusively
(e.g., through controlled studies) that a change is
an effect of the intervention; it is sufficient to show
that the change might reasonably be expected to
result from the intervention.
• It is unnecessary to undertake data analysis to
correct for secular trends (e.g., changes that
reflect continuing growth or decline in a measure
because of external forces over an extended
period). The measured improvement should
reasonably be determined to have resulted from
the intervention.
9.4 Is there statistical evidence (e.g., significance
tests) that any observed improvement is the result of
the intervention?
9.5 Was sustained improvement demonstrated
through repeated measurements over time?
9.6 Overall assessment: In the comments section,
note any recommendations for improving the
significance and sustainability of improvement as a
result of the PIP.

62 | PROTOCOL ONE

Yes

Comments

Worksheet 1.10. Perform Overall Validation of PIP Results
Instructions. Provide two overall validation ratings of the PIP results. The first rating refers to the EQRO’s overall
confidence that the PIP adhered to acceptable methodology for all phases of design and data collection and
conducted accurate data analysis and interpretation of PIP results. The second rating refers to the EQRO’s overall
confidence that the PIP produced evidence of significant improvement. Insert comments to explain the ratings.
Provide comments to justify the ratings.
Acronyms: EQRO = External Quality Review Organization; PIP = Performance Improvement Project.
PIP Validation Ratings (check one box)

Comments

Rating 1: EQRO’s Overall Confidence that the PIP
Adhered to Acceptable Methodology for All Phases
High confidence
Moderate confidence
Low confidence
No confidence
Rating 2: EQRO’s Overall Confidence that the PIP
Produced Evidence of Significant Improvement
High confidence
Moderate confidence
Low confidence
No confidence

PROTOCOL ONE | 63

Worksheet 1.11. Framework for Summarizing Information about PIPs
Instructions. To assist with the analysis portion of the EQR technical report requirement, complete the table below in its entirety for all PIPs. Doing so ensures the
EQROs generates comparable information for all PIPs. These tables can be included within an EQR technical report as a PIP appendix.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review Organization; LTSS = Long-Term
Services and Supports; MCP= Managed Care Plan; NQF = National Quality Forum; PIP = Performance Improvement Project.

1. General PIP Information
MCP name:
PIP title:
PIP aim statement:
Was the PIP state-mandated, collaborative, statewide, and/or plan choice? (check all that apply)
State-mandated (state required MCPs to conduct a PIP on this specific topic)
Collaborative (MCPs worked together during the planning or implementation phases)
Statewide (the PIP was conducted by all MCPs within the state)
Plan choice (state allowed the MCPs to identify the PIP topic)
Target age group (check one):
Children only (ages 0–17)*

Adults only (age 18 and over)

Both adults and children

*If PIP uses a different age threshold for children, specify the age range here:
Target population description, such as beneficiaries who are dually eligible for Medicare and Medicaid, LTSS, or Pregnant Women (please specify):
Programs:

Medicaid (Title XIX) only

64 | PROTOCOL ONE

CHIP (Title XXI) only

Medicaid and CHIP

2. Improvement Strategies or Interventions (changes tested in the PIP)
Enrollee-focused interventions (enrollee interventions are those aimed at changing enrollee practices or behaviors, such as financial or non-financial
incentives, education, and outreach):

Provider-focused interventions (provider interventions are those aimed at changing provider practices or behaviors, such as financial or nonfinancial incentives, education, and outreach):

MCP-focused interventions/system changes (MCP/system change interventions are aimed at changing MCP operations; they may include new
programs, practices, or infrastructure, such as new patient registries or data tools) :

3. Performance Measures and Results (add rows as necessary)
Performance
measures (be specific
and indicate measure
steward and NQF
number if applicable):

Baseline
year

Baseline
sample size
and rate

Most recent
remeasurement year
(if applicable)
Not Applicable—PIP is in
planning or implementation
phase, results not available

Most recent
remeasurement
sample size and rate
(if applicable)

Demonstrated
performance
improvement
(Yes/No)

Statistically significant
change in performance
(Yes/No)

Yes

Specify P-value
No

Specify P-value:
<.01

<.05

Other (specify):
Not Applicable—PIP is in
planning or implementation
phase, results not available

Yes
No

Yes

Specify P-value:
<.01

<.05

Other (specify):

PROTOCOL ONE | 65

4. PIP Validation Information
Was the PIP validated?

Yes

“Validated” means that the EQRO reviewed all relevant parts of each PIP and determined its validity. In many cases, this will involve calculating a score for each relevant stage of the
PIP and providing feedback and recommendations.

Validation phase (check all that apply):
PIP submitted for approval
First remeasurement

Planning phase

Implementation phase

Second remeasurement

Other (specify):

Baseline year

Validation rating #1: EQRO’s overall confidence that the PIP adhered to acceptable methodology for all phases of design and data collection, conducted
accurate data analysis and interpretation of PIP results,
High confidence

Moderate confidence

Low confidence

No confidence

Validation rating #2: EQRO’s overall confidence that the PIP produced significant evidence of improvement.
High confidence

Moderate confidence

Low confidence

EQRO comments on validation ratings:
EQRO recommendations for improvement of PIP:

66 | PROTOCOL ONE

No confidence

Worksheet 1.12. PIP Validation Reporting
Instructions. For each required validation rating, indicate the finding and provide comments to justify the rating.
Include recommendation for improvement, if applicable. If a PIP receives a “No confidence” validation rating, explain
the rationale and offer specific recommendations for improvement. PIPs rated “No confidence” should not be
continued or repeated without implementing recommended improvements. If a PIP is still in progress and a rating
cannot yet be assigned for the PIP improvement strategies, select “Not Applicable” and add a comment
explaining why.
The table below provides example validation scoring approaches for assessing both the methodology and
implementation of the PIP and the effectiveness of the PIP improvement strategy. If the EQRO prefers a different
rating methodology, use Section 2 of this worksheet to provide those definitions.
PIP methodology and implementation
•

High confidence. The PIP:

PIP improvement strategy
•

High confidence. The PIP produced evidence
for significant, sustained improvement as
evidenced by repeated measurements

•

Moderate confidence. The PIP demonstrated
some performance improvement, but:

○ Adhered to an acceptable methodology for all design
and data collection phases and,
○ Conducted accurate data analysis and interpretation of
PIP results
•

○ Evidence of improvement was inconsistent
or not sustained across measurement
periods and/or

Moderate confidence. The PIP:
○ Generally adhered to an acceptable methodology, but
with minor weaknesses or inconsistencies in the design
or data collection phases and/or
○ Conducted data analysis and interpretation that was
mostly accurate but may have small flaws or gaps that
slightly limit the reliability or generalizability of findings

•

○ Improvement did not reach statistical
significance, but trends suggest a potential
positive impact
•

Low confidence. The PIP:

○ Results that were not statistically significant
and lacked clear trends and/or

○ Demonstrated significant weaknesses or
inconsistencies in adhering to an acceptable
methodology for design or data collection phases
and/or
○ Conducted data analysis or interpretation with notable
errors, omissions, or assumptions that undermine the
reliability of results or make findings less actionable
•

No confidence. The PIP:
○ Did not adhere to an acceptable methodology for
design or data collection phases with critical flaws that
invalidate the process and/or

Low confidence. The PIP showed minimal or
inconsistent improvement with

○ Evidence of improvement was temporary or
did not persist across measurement periods
•

No confidence. The PIP provided no
evidence of meaningful improvement with
○ Performance remaining stagnant or
worsening, and/or
○ No statistically significant results or positive
trends

○ Conducted inaccurate data analysis or interpretation,
rendering the results unreliable, misleading, or
unusable for improvement activities
While this template encompasses CMS’s expectations for what EQR technical reports share when reporting PIP
validation findings, using this template is voluntary. This template has a companion Excel workbook. It can be
included in the EQR technical report or submitted to CMS as a separate document. If submitted separately, the
document must be posted to the state’s website.
Acronyms: MCP= Managed Care Plan; PIP = Performance Improvement Project; EQRO = External Quality Review
Organization

67 | PROTOCOL ONE

Overall PIP Validation Ratings

MCP

PIP

PIP methodology and
implementation
validation rating

PIP improvement
strategies validation
rating

High confidence

Moderate confidence

Low confidence

No confidence

EQRO recommendations

Not applicable—PIP
is in the planning or
implementation phase,
and results not available
High confidence

High confidence

Moderate confidence

Low confidence

No confidence

No confidence
Not applicable—PIP
is in the planning or
implementation phase,
and results not available

PIP Validation Rating Definitions (if applicable)
Validation rating
PIP methodology and implementation

PIP improvement strategies

68 | PROTOCOL ONE

Description

EQRO comments

Example of Worksheet 1.12. PIP Validation

MCP
Plan A

PIP
Clinical PIP:
Comprehensive
Diabetes Care

PIP methodology and
implementation validation
rating

PIP improvement
strategies validation
rating

High confidence

Moderate confidence

Low confidence

No confidence

EQRO recommendations

EQRO comments

•

Introduce real-time monitoring
tools like clinical dashboards to
track key indicators

•

Establish regular feedback loops
with providers to evaluate
implementation fidelity

Methodology for all phases of
design and data collection
adhered to acceptable
standards. Data analysis and
interpretation of results were
accurate. The interventions
produced some evidence of
improvement from baseline to
the first remeasurement, but
this was not sustained in the
second remeasurement
period.

Not applicable—PIP is
in the planning or
•
implementation phase,
and results not available

Non-clinical PIP:
Increase Enrollment
in the Wellness
Program

High confidence

Moderate confidence

Low confidence

No confidence

Monitor patient-specific barriers
to accessing care, such as
transportation to appointments
and health literacy

•

Incorporate evidence-based
diabetes care strategies such as
a medication management
program to support continued
progress

•

Conduct subgroup analyses by
stratifying data by race and
ethnicity or geography to identify
potential disparities

•

Develop a sustainability plan that Methodology for all phases of
design and data collection
includes periodic reviews to
adhered to acceptable
assess sustained progress
standards. Data analysis and
Standardize processes by
interpretation of results were
developing and documenting
accurate. Interventions were
clear workflows for outreach,
linked to the stated problem
enrollment, and engagement
and provided evidence of
Introduce periodic check-ins to
clinically significant and
re-engage enrollees who may
sustained improvement in
have dropped out
outcomes.

•

Not applicable—PIP is
in planning or
implementation phase,
•
results not available

PROTOCOL ONE | 69

Worksheet 1.13. PIP Summary Reporting
Instructions: Share each PIP’s aim, interventions (improvement strategies), target population(s), performance measure(s), and baseline and remeasurement
data, as available. If remeasurement data are unavailable, note “Not Available” and provide explanations for why data cannot be reported. In the “EQRO
comments” column, share observations on the PIP, such as promising practices that show potential for broader application or lessons learned from interventions
that had minimal impact. These comments can provide valuable insights into the effectiveness of various strategies, highlight opportunities for refinement, and
guide future quality improvement efforts. Colors, shading, or icons can be used to indicate how MCPs are performing relative to the state’s goal or benchmark for
a measure.
For PIPs in the planning phase during the 12 months before the finalization of the final report, include a note in the “EQRO comments” column stating that the
MCP PIP is still in the design phase and provide a brief description of the proposed focus and goals, if available. This note should clarify that the PIP is not yet
active but is expected to address [specific quality or performance areas] once implemented and provide an expected implementation date.
For states with multiple managed care programs, complete one table for each program. For states with multiple MCPs conducting PIPs on the same topic (e.g.,
diabetes care, infant well-child visits), complete one table with all MCPs.
While this template encompasses CMS’s expectations for what EQR technical reports share when reporting PIP findings, the use of this template is voluntary. This
template has a companion Excel workbook. It can be included in the EQR technical report or submitted to CMS as a separate document. If submitted separately,
the document must be posted to the state’s website.
Acronyms: MCP= Managed Care Plan; PIP = Performance Improvement Project.

MCP

PIP aim

PIP interventions

70 | PROTOCOL ONE

Target
population

Performance
measure

Baseline rate
(Baseline
Year)

Remeasurement
1 (Year)

Remeasurement
2 (Year)

EQRO comments

Example Worksheet 1.13. PIP Summary
MCP

PIP aim

PIP interventions

Target
population

Performance
measure

Baseline rate
(Baseline Year)

Remeasurement
1 (Year)

Ages 18-39
with type 1
or type 2
diabetes

Blood
Pressure
Control

68%

72%

(2023)

↑

Ages 18-39
with type 1
or type 2
diabetes

HbA1c
Testing

Remeasurement
2 (Year)

EQRO comments

Comprehensive Diabetes Care PIPs
Plan A

Increase Blood
Pressure Control to
73%

Provided glucose monitors
to enrollees, and related
education classes, to
improve self-management.

51%

42%

(2023)

↓

Passive nature of
this intervention
failed to engage
enrollees actively in
their care

(2024)

Sent performance reports
on diabetes care metrics to
providers to identify gaps
and drive improvement
Plan B

Providers reported
that access to
individualized
performance
reports helped
identify patients
who needed closer
follow-up care

(2024)

Sent performance reports
on diabetes care metrics to
providers to identify gaps
and drive improvement
Increase HbA1c
Testing to 74%

Providers reported
that access to
individualized
performance
reports helped
identify patients
who needed closer
follow-up care

Increase Blood
Pressure Control to
68%

Provided members with self- Ages 18-39
management education
with type 1
pamphlets
or type 2
diabetes

Blood
Pressure
Control

63%

64%

(2023)

↑

Increase HbA1c
Testing to 77%

Provided members with self- Ages 18-39
management education
with type 1
pamphlets
or type 2
diabetes

HbA1c
Testing

72%

(2023)

(2024)

Increase Eye Exam
Performed to 55%

Provided members with self- Ages 18-39
management education
with type 1
pamphlets
or type 2
diabetes

Eye Exam
(Retinal)
Performed

50%

52%

(2023)

↑

(2024)

PROTOCOL ONE | 71

MCP

PIP aim

PIP interventions

Plan C

Increase Blood
Pressure Control to
75%

PIP in the design phase

Increase HbA1c
Testing to 77%

PIP in the design phase

Target
population

Performance
measure

Baseline rate
(Baseline Year)

Remeasurement
1 (Year)

Remeasurement
2 (Year)

EQRO comments

Ages 18-39
with type 1
or type 2
diabetes

Blood
Pressure
Control

70%

Not applicable.

Ages 18-39
with type 1
or type 2
diabetes

HbA1c
Testing

72%

(2024)

PIP implementation
begins April 2024,
and results will be
included in the next
report
-

(2024)

Notes:
- indicates that an MCP does not have data for this remeasurement period.
↑ indicates that MCP's performance on the measure is improving.
↓ indicates that MCP’s performance on the measure is declining.

END OF WORKSHEETS FOR PROTOCOL 1

72 | PROTOCOL ONE

Not applicable.
PIP implementation
begins April 2024,
and results will be
included in the next
report

Protocol 2. Validation of Performance
Measures
A MANDATORY EQR-RELATED ACTIVITY

ACTIVITY 2: CONDUCT SITE VISIT ACTIVITIES
ACTIVITY 3: CONDUCT POST-SITE VISIT ACTIVITIES
ACTIVITY 4: REPORT RESULTS TO THE STATE

Background
States use performance measures to monitor the performance of individual
managed care plans (MCPs) at a point in time, to track performance over time,
to compare performance among MCPs, and to inform the selection and
evaluation of quality improvement activities. States specify standard
performance measures that the MCPs must include in their quality assessment
and performance improvement (QAPI) program. 42
States and MCPs often use measures included in the Centers for Medicare &
Medicaid Services (CMS) Child and Adult Core Sets to monitor and track the
quality of care in Medicaid and the Children’s Health Insurance Program
(CHIP). 43 While states' use of these measures for monitoring is voluntary, CMS
encourages states to adopt and use the Child and Adult Core Set measures to
support their managed care quality measurement and improvement initiatives.
Many Core Set measures are part of the Healthcare Effectiveness Data and
Information Set (HEDIS®) and have national and regional benchmarks.
As states continue to refine their quality improvement efforts, another key area
to consider is using performance data to address differences in access, quality,
and outcomes. Standardized data stratification is an important first step toward
improving the health of Medicaid and CHIP beneficiaries. To best monitor
differences in access, quality, and outcomes, states could consider stratifying

More information about QAPI and performance measure validation is available at 42 C.F.R.
438.330(b)(2) and (c), cross referenced by CHIP at 457.1240(b).

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/qualityof-care/performance-measurement/child-core-set/index.html. More information about the Adult Core
Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adultcore-set/index.html.

PROTOCOL TWO

ACTIVITY 1: CONDUCT PRE-SITE VISIT ACTIVITIES

performance measures by age, race, ethnicity, sex, geography, primary language, disability
status, or other demographics of interest. See Box 2.1 for considerations when stratifying
performance measures.
Box 2.1. Considerations when Stratifying Performance Measures
When planning to stratify performance measures by factors such as race, ethnicity, or other sociodemographic
characteristics, states and their EQROs should consider the following steps to ensure alignment with federal
guidance, data quality standards, and state priorities.
• Review the state's managed care quality strategy to identify the disparity factors into which the state would
like additional insights.
• Review the Office of Management and Budget standards for best practices on stratifying data for race and
ethnicity categories, available at https://www.federalregister.gov/d/2024-06469.
• Consider aligning the stratification approach with HEDIS®, mandatory Core Sets reporting, and Medicaid
and CHIP Quality Rating System requirements.
• Review CMS’s resource of health equity-related data definition, standards, and stratification practices for
suggested practices for sociodemographic elements, available at https://www.cms.gov/files/document/cms2024-omh-data-definitions.pdf.

Federal regulations at 42 C.F.R. 438.330(c) require states to specify standard performance
measures for MCPs to include in their comprehensive QAPI programs. 44 Each year, the MCPs
must (1) measure and report to the state the standard performance measures specified by the
state; (2) submit specified data to the state which enables the state to calculate the standard
performance measures; or (3) use a combination of these approaches.
This protocol is used to guide the validation of the performance measures specified by states for
inclusion in MCPs’ QAPI programs. It applies both when the QAPI performance measure is
calculated by the MCP and when it is calculated by the state. All QAPI measures must be
validated. However, the scope of this protocol is not limited to QAPI measures and can also be
applied to other performance measures as necessary.
In general, the external quality review organization (EQRO) must assess whether the
performance measures calculated by the MCPare accurate based on the measure specifications
and state reporting requirements (42 C.F.R. 438.330(b)(2)). 45 The state provides the list of
performance measures to be validated, the specifications for the measures, and the requirements
for reporting. The state must ensure that the EQRO is provided with a complete list of each

More information about QAPI and performance measures is available at 42 C.F.R. 438.330(b)(2). This is cross-referenced by
CHIP at 42 C.F.R. 457.1240(b).

While the protocol is written as if the MCP is calculating performance measures, the MCP may contract with another entity to
calculate and report on its behalf. Alternatively, 42 C.F.R. 438.330(c)(ii) allows the state to require the MCP to submit data to
the state, which the state then uses to calculate the performance measure. CHIP regulations cross-reference to these regulations
at 457.1240(b). This protocol applies in either circumstance.

74 | PROTOCOL TWO

MCP’s QAPI performance measures and that the EQRO validates each measure as part of the
performance measure validation activity.
As noted in the Introduction, states have the option to use information from a Medicare or private
accreditation review of an MCP to provide information for the annual external quality review
(EQR) instead of conducting this mandatory EQR-related activity. 46, 47 Should the state have its
EQRO leverage this information, such as HEDIS® Compliance Audits, the EQRO must review
the validation findings and validated performance measure data and include these results and
data in the EQR technical report.
A related protocol, Protocol 7, may be used by EQROs to calculate additional performance
measures in accordance with state specifications.

Getting Started on Protocol 2
To complete this protocol, the EQRO must undertake four activities (Figure 2.1, next page), all
of which should be completed within the 12 months preceding the finalization of the annual EQR
technical report. 48 49 The validation process is interactive and concurrent with MCP performance
measure calculation. Each phase is designed to support activities that may be conducted on-site
or remotely.
,

If the state elects to use nonduplication for this mandatory EQR-related activity (42 C.F.R. 438.360 (Nonduplication of mandatory
activities with Medicare or accreditation review), then the state must ensure that all information from the Medicare or private
accreditation review is provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42 C.F.R.
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an option). Use of nonduplication
must be identified in the state’s quality strategy (see 42 C.F.R. 438.360(c) and 438.340(b)(10)). CHIP cross-references to this
requirement at 42 C.F.R. 457.1250, but does not allow for the use of Medicare review activities for nonduplication.

A state may not utilize nonduplication if Medicare has accepted only an attestation of a plan’s quality improvement program
(QIP). In the context of this EQR-related activity, the QIP would have to undergo validation as part of a Medicare review for
nonduplication to be an option. See 42 C.F.R. 438.360(a)(2).

48 The “12 months preceding” refers to the timeframe during which the EQRO conducts the EQR activity prior to posting the

annual EQR technical report. This period is distinct from the 12-month data period subject to review, which reflects the most
recently concluded contract or calendar year being evaluated. See Box 6 in the Introduction for additional information.

49 If onsite activities are not feasible, site visits may be conducted virtually.

PROTOCOL TWO | 75

Figure 2.1. Protocol 2 Activities

Two supplemental resources are available to help EQROs validate performance measures:
•

Worksheets for Protocol 2. Performance Measurement Validation Tools, which can be used
to prepare for, conduct, and summarize the performance measure validation

•

Appendix B. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

The remainder of this protocol outlines the steps associated with Activities 1 through 4.

76 | PROTOCOL TWO

Activity 1: Perform Preliminary Review (Pre-Site Visit)
Step 1: Define the Scope of the Validation
The performance measures each state requires will depend
on the state's specific needs. In this step, the state provides
the EQRO with a list of all the MCP QAPI performance
measures, as well as any additional measures the state
requires, along with requirements for data collection and
reporting (e.g., sampling guidelines and instructions for
calculating numerators and denominators). This activity
should begin from two to six months before the
planned visit.
The EQRO should use Worksheet 2.1 to enumerate the
performance measures to be validated under Protocol 2,
including their data source, reporting frequency, and
format. Five data sources are used to produce MCP
performance measures:

WORKSHEET 2.1
WORKSHEET 2.2
Resources for Activity 1, Step 1
Worksheet 2.1. List of Performance
Measures to be Validated
• Provides a template for identifying the
measures the EQRO will validate for
the state, including the source, how
frequently to calculate each measure,
and when each measure is due to the
state
Worksheet 2.2. Performance Measure
Validation Template
• Provides a template for documenting
audit specifications for the validation
components of each performance
measure listed in Worksheet 2.1.,
and to assess the MCP’s
measurement and reporting process
for each component

Administrative data, such as claims/encounter data,
registries, or vital records.

Medical record review.

Administrative data supplemented by medical record
review, referred to as the “hybrid” method.

Electronic health records (EHRs).

Surveys (survey administration and validation is addressed in Protocol 6).

For each measure to be validated, the EQRO should complete Worksheet 2.2 (or a similar tool).
The worksheet is used to systematically gather information about the validation components and
audit specifications based on existing documentation about the measure. Elements include:
•

Documentation related to the data collection and calculation method.

•

Denominator calculation(s), including adequacy of the data sources to calculate the
denominator, operationalization of the measure-specific eligibility criteria, and adherence to
the measurement period.

•

Numerator calculation(s), including adequacy of the data sources to calculate the numerator,
appropriateness of codes used to identify numerator compliance, avoidance of double
counting, and adherence to the measurement period.

•

Sampling methodology (if used).

PROTOCOL TWO | 77

•

Reporting of rates and other supporting information, including documentation of deviations
(if any).

Worksheet 2.2 also contains an example of a completed performance measure validation
worksheet. The illustrative template is for Chlamydia Screening in Women (CHL-CH, Measure
Steward: National Committee for Quality Assurance, National Quality Forum [NQF] # 0033),
which is included in both the Child and Adult Core Sets and calculated using the administrative
method. During Activity 1, Step 1, the EQRO should begin to populate the audit specifications
based on the available measure documentation. Note that the worksheet is intended to serve as a
“living document” for the measure validation process, and the EQRO can adapt the template if
necessary.

Step 2: Assess the Integrity of the MCP’s Information System
This step helps focus the validation activities on aspects of
the MCP’s information system that are most likely to be an
issue in the validation process. Before validating individual
performance measures, the EQRO must assess (1) the
integrity of the MCP’s information system, (2) the
completeness and accuracy of the data produced, and (3) the
readiness of the MCPs’ data systems for calculating
performance measures. As part of this step, the EQRO
conducts an Information Systems Capabilities Assessment
(ISCA) for each MCP, as described in the
following sections.
Conduct an ISCA
Before conducting the site visit, the EQRO should provide
the MCP with information on the ISCA process, including
Worksheet B.1. The ISCA is used to validate MCP
information systems, processes, and data. The ISCA
corresponds to the objectives identified in this protocol and
addresses key components of calculating performance
measures including:
•

General information about the MCP.

•

Membership/enrollment data systems.

•

Claims/encounter data processing.

•

Provider data.

•

Data completeness.

•

Integration of data for performance measure calculation.

78 | PROTOCOL TWO

WORKSHEET B.1
WORKSHEET B.2
Resources to Conduct an ISCA
The ISCA is used to validate MCP
information systems, processes, and
data. The ISCA provides a foundation
for the validation of performance
measures.
Appendix B ISCA Capabilities
Assessment
• Explains how to conduct the ISCA
Worksheet B.1. ISCA Tool
• Provides a template for MCPs to
document the capabilities of the
information systems, processes, and
data
Worksheet B.2. ISCA Interview Guide
• Provides a guide to EQROs for
conducting follow-up interviews with
MCP staff to record responses and
document specific issues based on
findings from Worksheet B.1

The ISCA provides information about the timing of any other recent, independent, documented
assessment, such as a HEDIS® Compliance Audit™. If the MCP recently had a comprehensive,
independent assessment of its information systems, the EQRO may review those results. If the
MCP has not had an ISCA within a time frame determined by the state, 50 the EQRO will conduct
an ISCA as part of this protocol. It is recommended that EQROs request that MCPs provide any
assessments of their information technology (IT) systems conducted in the previous two years.
The EQRO should document the strengths and weaknesses of the MCP information system
relevant to the types of data used by the MCP in calculating performance measures. The EQRO
should take into account systems issues (such as missing data), when validating individual
performance measures and determining whether they are reportable.
Assess MCP Data Systems and Types
The EQRO should assess every data system and type of data the MCP processes to ensure the
required data are current and accurate, particularly at the time it extracts data for its performance
measures. The EQRO should assess changes in the MCP’s data systems that might affect the
production of the performance measures. Major changes, upgrades, or consolidations within the
system, or acquisitions/mergers with other MCPs may impact the accuracy or completeness of
required data elements. Elements that should be assessed for each MCP data system and
type include:
•

Membership/enrollment data.

•

Provider data.

•

Claims and encounter data.

•

Medical records data.

•

Pharmacy, laboratory, and other ancillary data .

Membership/Enrollment Data
The EQRO should assess:
•

The MCP’s ability to track enrollees over time, changes in enrollment, name changes, and
changes in coverage.

•

The MCP’s processes to ensure membership/enrollment data are current and accurate.

•

Changes in the MCP’s membership data systems that might affect the production of the
performance measures.

There is no statutory or regulatory requirement for the frequency with which ISCAs should be conducted. Each state must
determine the maximum interval between assessments of MCP information systems, balancing the cost to the state and burden
on the MCP with the need to ensure that changes to the MCP’s information systems are assessed frequently enough to support
accurate performance measurement.

PROTOCOL TWO | 79

•

Whether transactions between the MCP and state data systems (such as state eligibility files)
affect measure calculation through updating, correcting, or overwriting source data (e.g., race
or ethnicity information).

The EQRO should also determine whether each MCP enrollee is uniquely identifiable and can be
linked to the state’s Medicaid and CHIP eligibility file. The membership/enrollment database
may capture the following information for every enrollee:
•

Unique enrollee identifier (ID), including state-issued Medicaid and CHIP ID and CMSissued Medicare number (if applicable).

•

Eligibility category.

•

Date of birth.

•

Sex.

•

Race and ethnicity.

•

Primary language.

•

Disability status.

•

Enrollment and/or termination dates, including multiple enrollment and termination dates
within and across programs (preferably exact dates rather than monthly indicators).

•

Primary care provider (e.g., provider name, provider ID number, provider location).

Collecting and assessing membership and enrollment data are increasingly important due to the
requirement that each state’s managed care quality strategy include a plan for identifying,
evaluating, and reducing health disparities based on age, race, ethnicity, sex, primary language,
and disability status (42 C.F.R. 438.340(b)(6)). Under this requirement, states must identify this
demographic information for each Medicaid enrollee and provide it to the MCP at the time
of enrollment.
In addition, to facilitate geographic stratification of performance (such as analyses of access and
timeliness of care), complete and accurate information on the household’s location of residence
(e.g., ZIP code) is also desirable.
Finally, to facilitate surveys of patient experience, complete contact information is essential. At a
minimum, name and address are required; phone numbers and email addresses are highly
desirable. See Protocol 6 for more information.
Provider Data
The EQRO should review an MCP’s provider data system(s) to assess the MCP’s ability to track
providers over time, across multiple office locations, and through changes in participation. In
addition, the EQRO should assess how many contracted providers use electronic health records

80 | PROTOCOL TWO

(EHRs) and the extent to which EHRs are used in the calculation of an MCP’s
performance measures.
Claims and Encounter Data
Claims and encounter data should cover all types of services offered by the MCP and not
separately contracted by the state, such as hospital inpatient, hospital outpatient, primary care,
skilled nursing facility, nursing facility (custodial care), specialty care, behavioral health care,
family planning services, home health care, radiology, laboratory, pharmacy, dental care, and
vision care. The EQRO should note the following for each type of claim/encounter data
captured:
•

Total number of diagnosis and procedure codes (such as Healthcare Common Procedure
Coding System (HCPCS) codes and Current Procedural Terminology (CPT)® codes, the
American Dental Association’s Common Dental Terminology (CDT)© codes, and
International Classification of Diseases (ICD)-10 Procedure Coding System codes), captured
by the system.

•

Whether the principal diagnosis, secondary diagnoses, and procedure codes can be accurately
distinguished in the system.

•

Maximum number of digits/characters captured for each data field in each type of claim or
encounter.

The accuracy and validity of measures may be adversely affected if the information system (IS)
truncates codes or is unable to collect and/or differentiate among a sufficient number of codes.
The EQRO should understand the various coding systems and forms used by the MCP and its
vendors to capture and process clinical information through its claims and encounter databases.
The EQRO should assess how well the IS translates or maps these codes back to the criteria for
MCP performance measure reporting, and how it ensures the accuracy of these
translation processes.
The EQRO should also determine, through review of existing documentation or in consultation
with the MCP, whether certain diagnosis or procedure codes required for performance
measurement are not accurately or completely captured in the claims and encounter data systems,
such as maternity or dental care, behavioral health care, and preventive care services.
Medical Record Data
The EQRO should use medical record data to review:
•

Methods used to retrieve information from medical records.

•

Training and tools that medical record review staff receive.

PROTOCOL TWO | 81

•

Processes used to ensure accurate data retrieval, inter-rater reliability, and data entry into a
database used to produce performance measures.

With increasing adoption of EHRs and state use of Health Information Exchanges (HIEs), MCPs
and provider practices may use newer methods to extract information from the medical record.
As noted earlier, the EQRO should assess how electronic records are used in performance
measure calculation and whether there are any special considerations in the validity and
reliability of these records for accurate measurement.
Pharmacy, Laboratory, and Other Ancillary Data
Pharmacy data use standardized codes for prescription drugs such as those promulgated by the
National Council for Prescription Drug Programs (NCPDP) 51 Laboratory services frequently use
a similar, nationally recognized system of coding (known as LOINC®). 52
Due to the diversity in the size, type, and ownership of pharmacies, laboratories, and other
ancillary providers, non-standard codes should be examined. When found, the EQRO should
assess the MCP’s system for cross-walking these codes to store the necessary information in its
performance measure database. The EQRO should understand the MCP’s mapping system of
non-standard codes to standardized codes and the mechanism used to ensure the accuracy of
these translation processes.
If the MCP does not collect pharmacy, laboratory, or other ancillary data through an
administrative or claims database, it may retrieve these data from medical records. However,
medical records are often unreliable due to non-standard coding and terminology, poor
coordination of records, and insufficient record linkages between primary care and specialist
providers. These issues should be addressed during the claims/encounter data review and the
medical record review and, if necessary, reflected on any corrective action plan.
The EQRO must assess the ability of the information system to link these different data sources.
For example, to identify enrollees with diabetes, an MCP may need to combine diagnosis code
data from inpatient or ambulatory encounters (not all ongoing conditions are reported at every
encounter) with pharmacy data, lab data, and/or a disease registry, an MCP’s disease
management system, or a medical management system used by MCP staff, if one exists. Thus, to
determine whether enrollees with diabetes have received a retinal examination from an
ophthalmologist or optometrist within the previous year, the MCP would have to link diagnosis
and procedure code data from encounter forms, medical records, and/or claims data with
information about the specialty of the providers that performed the examinations for
these enrollees.

More information about NCPDP codes is available at https://www.ncpdp.org/.

More information about LOINC codes is available at https://loinc.org/.

82 | PROTOCOL TWO

Synthesis of Findings
The EQRO will review the findings from the ISCA across each of the data systems and types.
The EQRO should note any problem areas related to the adequacy of the MCP’s data systems to
calculate and report the required performance measures. Where a response is incomplete,
indicates an inadequate process, or requires clarification, the EQRO should flag the issue for
follow-up and further review during the site visit.

Step 3: Conduct Detailed Review of Measures
In this step, the EQRO conducts a detailed review of
measures, incorporating findings from the ISCA. In its
detailed review, the EQRO should identify measures most
vulnerable to inaccurate results based on its knowledge of
the MCP’s data systems and processes. For example, if
the MCP uses global billing for maternity care, the
calculation of maternity measures could be affected by the
lack of separate claims for prenatal and postpartum care,
and thus, performance measurement results for such
measures could be significantly under-reported. Similarly,
the EQRO should identify certain types of claims that
may require linkage from other data sources (such as
laboratory, behavioral health, or dental) because the
necessary codes may not be available for all enrollees.
The detailed review of each measure involves a
systematic assessment of the code and output to assess
adherence to the specifications and the impact of any
systems issues on the accuracy and completeness of the
data. In addition, the EQRO should pay special attention
to issues frequently encountered in developing its audit
specifications based on findings from the ISCA:

WORKSHEET 2.2
Resource for Detailed Review of
Measures
A detailed review of each measure
includes the following:
• Source code for the measure
• Data mapping, if applicable
• Measure workflow
• Data output at each stage of the
measure calculation
• Record-level numerator and
denominator data
Worksheet 2.2. Performance Measure
Validation Template
• Provides a template for documenting
audit specifications for the validation
components of each performance
measure listed in Worksheet 2.1.,
and to assess the MCP’s
measurement and reporting process
for each component

•

Claims-dependent denominators.

•

Complex continuous enrollment criteria.

•

Use of global billing.

•

Identification of live births (including linkage of mother and infant records).

•

Procedure codes that are infrequently billed by providers (such as developmental screening,
documentation of Body Mass Index [BMI], or BMI percentile in the medical record).

•

Ability to link claims and pharmacy data.

•

Identification of practitioner type (especially mental health providers).
PROTOCOL TWO | 83

•

Multiple numerator events.

•

Vendor-supplied data.

During the detailed measure review, the EQRO should develop targeted audit specifications for
each measure to account for potential systems issues. The EQRO should record its audit
specifications and interim findings on Worksheet 2.2 or a similar worksheet. Box 2.2 provides
additional information on validating HEDIS® measures calculated with HEDIS®certified software.
Box 2.2. Review of HEDIS® Measures Calculated with HEDIS®-certified Software
If the state requires HEDIS® measures and the MCP used HEDIS®-certified software to calculate the measures, the
EQRO does not need to review source code for those measures. However, the EQRO is required to verify that the
measures were calculated as specified by the software and that systems issues did not compromise the accuracy and
completeness of the performance measures. As an example, when an MCP pays for prenatal and postpartum care as
part of a bundled maternity care payment, HEDIS® measures may be calculated according to the specifications but
the rates may be significantly under-reported using administrative data due to the lack of separate claims for prenatal
and postpartum care. Thus, the EQRO is required to review and validate the accuracy and completeness of HEDIS®
measures based on findings from the ISCA. The EQRO should also share the validation findings and performance
rates for these measures in the EQR technical report.
Some performance measures requiring medical record review (i.e., hybrid measures) are audited by an NCQAcertified HEDIS Compliance Auditor. When a hybrid measure has undergone such an audit and the state has received
the final audit report and attestation, the EQRO is not required to independently revalidate the medical record review
results under Activity 2, Step 4. In these circumstances:
• The state must provide documentation. The state must provide the EQRO with the final audit report, attestation,
and any related findings or corrective action documentation.
• The EQRO must incorporate audit results into the EQR technical report. The EQRO must review these
materials and describe in the EQR technical report how the audit informed its validation conclusions, including any
noted limitations.
• The EQRO must identify audited measures. The EQR technical report must clearly indicate the measures
subject to NCQA-certified audit.

Step 4: Initiate Review of Medical Record Data Collection
The purpose of this step is to verify the accuracy of the
medical record review conducted by the MCP when
medical record data are used to calculate and report
performance measures. If a plan only uses administrative
data, this step is not necessary.

Resource for Activity 1, Step 4

To validate the integrity of the medical record review
processes, the EQRO conducts the validation in two
phases: the first phase assesses the initial implementation
of the process to allow corrections at an early stage; the

• Provides instructions for conducting
the medical record review and
worksheets to summarize reabstraction findings from the review
(Worksheet 2.3, Table 1) and to
record the impact of findings from the
review (Worksheet 2.3, Table 2)

84 | PROTOCOL TWO

WORKSHEET 2.3

Worksheet 2.3. Medical Review
Validation Template

second phase is a retrospective review of the accuracy of the medical record review abstraction
process.
Review of Implementation of Medical Record Review
During the early implementation of the medical record abstraction process, the EQRO will
confirm the following about MCP activities:
•

Selection of staff with appropriate experience and credentials.

•

Development of high-quality abstraction tools to collect the required information.

•

Provision of effective staff training about the review process.

•

Implementation of sound oversight procedures to assess reviewer performance (such as
validation of a sample of records or tests of inter-rater reliability).

The EQRO may review a convenience sample of records across measures to identify potential
problems for MCP correction. National Committee for Quality Assurance’s (NCQA’s) HEDIS®
Compliance Audit™ recommends selecting up to ten difficult-to-review measures and obtain
copies of at least two complete medical record review tools and charts per measure. If the state
requires fewer than 10 measures that rely on medical record data, the EQRO should conduct the
sample review for all medical record-dependent measures. Completing this step early in the
process allows the MCP to address identified issues and resolve them during the initial stages of
data collection.
Re-abstraction and Validation of Medical Record Review
The EQRO will conduct a retrospective medical record review for at least two measures that
include medical record review either alone or in combination with administrative data (known as
the hybrid method). The EQRO should target statistical validation to measures that are new,
complex, and dependent on the medical record data or those with previously identified issues.
For each measure, the EQRO will request a sample of 30 medical records with positive
numerator events and compare the completed abstraction information to the medical record to
determine the rate of agreement. If the agreement rate is less than 100 percent, the EQRO will
assess the degree of bias. Worksheet 2.3 provides a detailed description of the medical record
review process and validation tool. The EQRO should summarize findings for the MCP from the
medical record review validation, including error rates for the validated measures (see Table 2 in
Worksheet 2.3) and recommendations for improving the medical record review process.

PROTOCOL TWO | 85

Step 5: Prepare for the MCP Site Visit
In this step, the EQRO contacts the MCP, before
conducting the site visit to:
•
•

•

Explain the procedures and timeline for performance
measure validation activities.
Communicate the EQRO’s policies and procedures for
safeguarding confidential information and signed
confidentiality agreements.
Organize the site visit to ensure the availability of
necessary documentation and staff (see Box 2.3).

WORKSHEET 2.4
Resource for Activity 1, Step 5
Worksheet 2.4. Potential Documents
and Process for Review
• Provides a checklist of documents,
data, and procedures the MCP
should make available before or
during the site visit

Box 2.3. Potential Site Visit Participants
During the site visit, the MCP should arrange for staff and vendors to meet with the EQRO to provide information
about the processes to processes to calculate or report performance measures. The EQRO may want to suggest to
the MCP that corporate staff—particularly IS staff—be included in the site visit as corporate staff may provide
additional insight into some interview questions. Participants may include:
• The Director of Health/Medical Information Systems
• Information system programmers or operators
• Director of Member/Patient Services and staff
• Director of Utilization Management and staff
• Director of Quality Improvement and staff

At this stage, the EQRO should also request confirmation of the list and description of staterequired performance measures. The EQRO will provide the MCP a list of documents, data, and
procedures that may be reviewed before or during the site visit.

Activity 2: Conduct MCP Site Visit
Site visit activities provide an opportunity for the EQRO to follow up on findings from the presite IS assessment and to confirm or clarify information about the production and reporting of
performance measures through document review or discussions (see Box 2.4, next page).

86 | PROTOCOL TWO

Box 2.4. Purpose of the Site Visit
Site visits play a critical role in the performance measure validation process by allowing reviewers to directly assess
the data systems and processes used by managed care plans. The following activities illustrate the key purposes of a
site visit and how it supports a comprehensive validation review:
• Confirm, observe, and query systems used to produce performance measure results, including membership,
medical, pharmacy, provider, and other ancillary or supplemental data sources.
• Investigate and follow up on issues identified from the ISCA.
• Assess data integration and control procedures for accurate production of the performance measures.
• Assess data completeness.
• Confirm processes for calculating and reporting the performance measures.

During the site visit, whether onsite or virtual, the EQRO will complete the following steps,
which are described below:
1.

Review the IS underlying performance measurement.

Assess data integration and control for performance measure calculation.

Review performance measurement production.

Complete the detailed review of measures.

Assess the sampling process.

Communicate preliminary findings and outstanding items.

Step 1: Review IS Underlying Performance Measurement
The review of the ISCA, which had begun during the presite phase, continues during the site visit. During this step,
the EQRO reviews the IS components that the MCP uses
to produce performance measures via (1) staff interviews,
(2) primary source documents, (3) systems and processes
used to calculate performance measures, (4) data entry
observation, and (5) data files. These sources are
described below.
Staff Interviews

WORKSHEET 2.5
Resource for Activity 2, Step 1
Worksheet 2.5. Interview Guide for MCP
Data Integration and Control Personnel
• Provides a list of interview questions
for key staff involved in the
production of performance measures
using questions tailored to the MCP’s
processes for producing these
measures. Tailor the questions as
appropriate

The EQRO will interview key staff (scheduled and
confirmed ahead of the visit) involved in producing
performance measures using questions tailored to the MCP’s processes for producing
performance measures based on findings from the ISCA. These interviews also provide an
opportunity to supplement the review of information system policies, procedures, and data
(described below).

PROTOCOL TWO | 87

Primary Source Verification
The EQRO will review the primary source documents, including paper forms and other input to
the MCP systems, and confirm that the information from the primary source matches the
information used for performance measurement. In addition, the EQRO will review the processes
used to input, confirm entry, and identify errors and processes used to transmit and track the data
through systems. Typical forms the EQRO will review include:
•

Enrollee-initiated enrollment data.

•

Hospital claims/encounters.

•

Ambulatory claims/encounters.

•

Prescription data.

•

Practitioner demographic forms.

•

Practitioner credentialing forms.

•

Claims logs.

•

Lab results.

System and Process Review
The EQRO will review the MCP’s documentation describing the systems and processes used to
calculate performance measures to confirm that they adhere to state policies and procedures.
These include systems and processes for collecting, storing, and reporting data. All
documentation received and examined must be recorded.
Observation
The EQRO will observe key MCP processes required for performance measure calculation to
assess data entry and other data manipulations. Examples include:
•

Data entry of membership updates, claims/encounter data, and practitioner data (e.g., confirm
that mandatory fields are required and invalid data elements are identified, such as invalid
birth dates or invalid service dates).

•

Claims operations, including overrides or exceptions.

•

Computer operations and security plans to confirm procedures are followed.

The EQRO will directly observe the Extract, Transform, and Load (ETL) process 53 and its
replication by two separate operators through the process using an observation guide to confirm

ETL is when these three database functions (extract, transform, and load) are combined into one tool to pull data out of one
database and place it into another database.

88 | PROTOCOL TWO

the activities, as well as the process where data are incomplete (e.g., a claim without a provider
identification number).
Data File Review
The EQRO will directly examine data files to confirm that the data are stored and processed
according to the documentation provided. Examples of files to review include:
•

Transaction files for clinical services, membership, and practitioner changes.

•

Intermediate files created by extracts, queries, and analysis applications.

•

Data repository files.

Step 2: Assess Data Integration and Control for Performance Measure
Calculation
In this step, the EQRO assesses the MCP’s ability to link
data from multiple sources and the extent to which the
MCP has created systems and processes to ensure the
accuracy of the calculated performance measures.
Worksheet 2.6 helps the EQRO review:

WORKSHEET 2.6
Resource for Activity 2, Step 2
Worksheet 2.6 Data Integration and
Control Findings Tool

•

Accuracy of data transfers to the assigned performance
measure repository.

•

Accuracy of file consolidations, extracts, and
derivations.

•

Adequacy of the performance measure data repository to calculate and report performance
measures.

•

Management of report production and reporting software.

• Guides the EQRO’s review of data
integration and control elements
during the site visit

PROTOCOL TWO | 89

Step 3: Review Performance Measure Production
In this step, the EQRO reviews the MCP’s documentation
of the entire process undertaken in the production of the
performance measures, including:
•

•
•

Data collection from various sources (e.g.,
membership, enrollment, provider, claims, or
encounter files; medical records; laboratory,
pharmacy, or other ancillary records).
Steps taken to integrate the required data into a
performance measure data set or repository.
Procedures or programs to query the data
set/repository to identify denominators, generate
appropriate samples, determine numerators, and apply
proper algorithms to the data to produce valid and
reliable performance measures.

WORKSHEET 2.7
WORKSHEET 2.8
Resources for Activity 2, Step 3
Worksheet 2.7. Data and Processes
Used to Produce Performance
Measures: Documentation and Review
Checklist
• Helps the EQRO check the
documentation of steps taken in the
production of the performance
measures
Worksheet 2.8. Data and Processes
Used to Produce Performance
Measures: Findings
• Provides a template for recording
findings based on measurement
plans, policies, and programming
specifications

90 | PROTOCOL TWO

Step 4: Complete the Detailed Review of Measures
In this step, the EQRO determines the extent to which the
MCP correctly used the technical specifications to produce
accurate performance measure results. All validation
components should be addressed during this step using
Worksheet 2.2 (or similar tool).
To ensure the integrity and comparability of the
performance measures, the EQRO should pay special
attention to factors affecting the accuracy and
completeness of the denominators and numerators. For
example, the EQRO should assess whether the MCP used
the appropriate data and methods to identify the entire
eligible population for the denominator (including linkage
of data from separate sources, application of inclusions
and exclusions, and creation of complex episodes, where
applicable). In addition, the EQRO should determine
whether the MCP correctly identified and assessed
qualifying medical events for the numerator to include all
appropriate events, while excluding events that do not
qualify. The EQRO should determine whether the
numerators and denominators were calculated
appropriately based on all applicable codes (such as
diagnoses, procedures, and prescription drugs) and all
available data sources (such as membership/enrollment
data, claim/encounter data, provider data, utilization or
medical management information systems data, or data
extracted from medical records).
For performance measures requiring medical record
review, the EQRO should validate the results of the
medical record review for 30 enrollees who met the
numerator requirements for at least two measures. For
more information, refer to Activity 1, Step 4.

WORKSHEET 2.2
WORKSHEET 2.9
WORKSHEET 2.10
WORKSHEET 2.11
Resources for Activity 2, Step 4
Worksheet 2.2. Performance Measure
Validation Template
• Provides a template for documenting
audit specifications for the validation
components of each performance
measure, and to assess the MCP’s
measurement and reporting process
for each component
Worksheet 2.9. Policies, Data, and
Information Used to Produce Measures:
Review Checklist
• Provides a checklist to tracks
documents and data used to assess
the accuracy of the MCP’s
performance measure calculations
Worksheet 2.10. Measure Validation
Findings
• Provides a template for documenting
adherence to denominator
specifications; programming logic,
source code, and calculations;
identifying medical events; exclusion
criteria; population estimates;
identifying the at-risk population;
inclusion of qualifying events in the
numerator; and medical record data
in the numerator
Worksheet 2.11. Interview Guide for
Assessing Processes and Procedures
Used to Produce Numerators and
Denominators
• Provides a list of interview questions
that can be tailored to supplement
findings recorded in Worksheet 2.10
For performance measures requiring
medical record review, use Worksheet
2.3. Medical Record Review Validation
Tool.

PROTOCOL TWO | 91

Step 5: Assess the Sampling Process (if applicable)
In this step, if applicable, the EQRO determines whether
the sample represents the entire eligible population in all
relevant dimensions. The MCP’s sampling method
should not exclude any population subgroups to which
the performance measure applies. For example, when
assessing well-child care, the sample should not exclude
children with special health care needs whose primary
care provider is a specialist other than a pediatrician or
family practitioner.

Step 6: Communicate Preliminary Findings
and Outstanding Items
At the conclusion of the site visit, the EQRO will
communicate preliminary findings to the MCP, including
any outstanding items for follow-up. The information
communicated during the closing conference will appear
in the EQRO’s subsequent preliminary report to the
MCP. In addition, the EQRO should provide a list of
outstanding items before completing the preliminary
report to allow the MCP the maximum time to resolve
identified issues.

WORKSHEET 2.12
WORKSHEET 2.13
Resources for Activity 2, Step 5
Worksheet 2.12. Policies, Procedures,
and Data Used to Implement Sampling:
Review Checklist
• Guides this review by providing a list
of documents, data, and procedures
to assess the sampling process
Worksheet 2.13. Sampling Validation
Findings
• For each measure involving a
sample, this worksheet helps assess
the extent to which:
○ The MCP followed the specified
sampling method to produce an
unbiased sample representative of
the entire included population
○ The MCP maintains its
performance measurement
population sample frame to allow
for a sample to be re-drawn or
used as a source for replacement
○ Sample sizes collected conform to
the methodology in the
performance measure
specifications
○ The sample is representative of
the entire population
○ Proper substitution methodology is
followed for performance
measures that include medical
record reviews

92 | PROTOCOL TWO

Activity 3: Perform Overall Validation (Post-Site Visit)
Following the site visit, the EQRO will:
•

•

Analyze all data and submit a preliminary report to the
MCP detailing areas of concern, suggested methods for
correction, and a timeline for the MCP to make
corrections,
Re-validate selected performance measures, and the
measurement processes the MCP used to make
corrections,

WORKSHEET 2.3
WORKSHEET 2.6
WORKSHEET 2.10
WORKSHEET 2.13
Resources for Activity 3
Information gathered in Activities 1 and
2 using the following worksheets and
tools may be helpful when preparing the
final validation report:

•

Re-evaluate the corrected information and submit a
report of validation findings to the state,

•

Determine preliminary validation findings for each
measure, and

•

Assess the accuracy of MCP’s performance measure
reports to the state.

• Describes procedures and sample
tools for validating medical review
findings

Note that throughout this EQR-related activity or during
any part of an EQR, the state may decide that immediate
corrective action is required.

Worksheet 2.6. Data Integration and
Control Findings Tool

Step 1: Determine Preliminary Validation
Findings for Each Measure

Worksheet 2.3. Medical Record Review
Validation Template

• Provides a template for recording
findings from interviews with data
integration and control MCP
personnel
Worksheet 2.10. Measure Validation
Findings

In the preliminary validation findings report, the EQRO
• Provides a template for recording
will document findings, identify areas of concern, and
findings from the measures record
make suggestions for corrective action or long-term
validation review
improvement for each of the performance measures the
Worksheet 2.13. Sampling Validation
Findings
EQRO validated. The report should indicate which MCP
performance measures and elements of the measures were
• Provides a template to record
findings from the sampling
invalid and, therefore, should not be reported (if any). The
assessment process
report should also provide the MCP with correctional
guidance for improving the overall measure production
process. In addition to communicating written findings, the EQRO may participate in meetings
with key MCP personnel responsible for calculating and reporting performance measures to
assist the MCP with implementing recommended corrective action.
Once the EQRO has submitted its preliminary findings to the MCP, the MCP may offer
comments and documentation to correct errors and omissions in the EQRO’s preliminary report.
The MCP may recalculate performance measures at the state's discretion based on the findings.
PROTOCOL TWO | 93

The EQRO must then revalidate the revised performance measure(s) and incorporate the MCP’s
comments or revised performance measure validation findings. If the state chooses not to allow
measure re-validation, the recommendations will be reviewed in the following year as part of the
MCP assessment of progress toward recommended improvements.

Step 2: Assess and Document the Accuracy of Performance Measure Reports
The EQRO will assess and document the extent to which the MCP reported the calculated
performance measures correctly in its final report to the state and verify the reporting of each
performance measure by reviewing the following:
•

Procedures for submitting reports that meet state requirements (such as specified format,
supporting documentation, and timing).

•

Documentation that the MCP appropriately implemented procedures to properly submit
required reports to the state.

The EQRO will always use the state’s decision rules for determining the degree to which each of
the MCP’s reported performance measures is accurate and complete. The decision rules for
compliance should be consistent across MCPs within the state.

Activity 4: Report Results to the State
In this activity, the EQRO reports its findings to the state, and
the state will submit the final technical report to CMS. The final
report should follow the state’s required format and include the
following elements:

WORKSHEET 2.14
WORKSHEET 2.15
WORKSHEET 2.16

•

A list of the measures validated by the EQRO.

•

A list of non-QAPI measures validated by the EQRO.

Resources for Activity 4

•

A list of QAPI measures and information from a Medicare
or private accreditation review to satisfy nonduplication.

Worksheet 2.14. Framework for
Summarizing Information About
Performance Measures

•

A description of the EQRO’s validation activities, including:

•

The EQRO team members involved in the validation.

• Provides a structure for summarizing
performance measure-level information

•

A summary of the validation strategy.

•

The data collection methods and analysis.

•

List of site visit participants (EQRO, MCP, and vendor).

• Provides a template for reporting
validation findings for multiple measures
and MCPs

•

Other considerations relevant to the site visit process.

Worksheet 2.16 Performance Measure
Reporting Template

•

Worksheets, tools, and other supporting documentation.

94 | PROTOCOL TWO

Worksheet 2.15. Performance Measure
Validation Reporting Template

• Provides a template for reporting
comparison performance measure data

•

Analyses and conclusions based on the validation process for each performance measure,
including the validation status of each performance measure (including the results of the
medical record review).

•

Findings on the MCP’s IS capabilities and data integration, including documentation of the
timing of the state’s most recent ISCA and a description of what documentation was
reviewed by the EQRO.

•

Performance measure results (not only the results of the validation), as well as the results
from any quantitative assessments. Box 2.5 shares additional guidance on reporting
performance measure results.

•

Recommendations for improving the process for calculating and reporting performance
measures, including implications for the MCP’s data systems, methods, and staffing (e.g.,
programming and analytic capacity).

When possible, the report should also identify recommendations from the previous year’s report
submitted to the state and discuss progress made on these recommendations over the past year
based on information gathered during the validation process. See “EQR Reporting” in the
Introduction for further guidance on producing a clear and concise report.
Box 2.5. Reporting Performance Measure Outcome Data and Quantitative Assessments Results
In the 2024 final rule, CMS clarified that for the mandatory validation activities, states must report outcomes data and
results from quantitative assessments in addition to validation findings For performance measure validation, this may
include quantitative analyses of managed care plan (MCP) performance using state-selected performance measures,
trends over time, and comparisons across MCPs and/or populations. Quantitative assessments may include analyzing
performance measure results relative to established targets or benchmarks; statistical analyses to determine whether
observed differences or changes are statistically significant; stratified analyses to assess variation across
subpopulations (e.g., eligibility groups, geography); and assessments of data completeness and validity for measures
used in performance monitoring.

END OF PROTOCOL 2

PROTOCOL TWO | 95

Worksheets for Protocol 2: Performance Measure Validation
Tools
Instructions. Use these or similar worksheets to assist in validating performance measures reported by the MCP.
These worksheets provide templates for validating performance measures and a framework for summarizing
information about performance measures in the EQR technical report. This tool includes the following worksheets
crosswalked to the applicable Activity and Step:
Worksheet Name

Protocol Activity and Step

Worksheet 2.1. List of Performance Measures to be
Validated

Activity 1. Step 1. Define the Scope of the Validation

Worksheet 2.2. Performance Measure Validation
Template

Activity 1. Step. 1. Define the Scope of the Validation
Activity 1. Step 3. Conduct Detailed Review of Measures
Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.3. Medical Review Validation Template

Activity 1. Step 4. Initiate Review of Medical Record Data
Collection
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.4. Potential Documents and Process for
Review

Activity 1. Step 5. Prepare for the MCP Site Visit

Worksheet 2.5. Interview Guide for MCP Data Integration Activity 2. Step 1. Review Information Systems
and Control Personnel
Underlying Performance Measurement
Worksheet 2.6. Data Integration and Control Findings
Tool

Activity 2. Step 2. Assess Data Integration and Control
for Performance Measure Calculation
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.7. Data Processes Used to Produce
Performance Measures: Documentation and Review
Checklist

Activity 2. Step 3. Review Performance Measure
Production

Worksheet 2.8. Data and Processes Used to Produce
Performance Measures: Findings

Activity 2. Step 3. Review Performance Measure
Production

Worksheet 2.9. Polices, Data, and Information Used to
Produce Measures: Checklist

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.10. Measure Validation Findings

Activity 2. Step 4. Complete the Detailed Review of
Measures
Activity 3. Perform Overall Validation (Post-Site Visit)

Worksheet 2.11. Interview Guide for Assessing
Processes Used to Produce Numerators and
Denominators

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.12. Policies, Procedures, and Data Used to Activity 2. Step 5. Assess the Sampling Process (if
Implement Sampling: Review Checklist
applicable)
Worksheet 2.13. Sampling Validation Findings

Activity 2. Step 5. Assess the Sampling Process (if
applicable)
Activity 3. Perform Overall Validation (Post-Site Visit)

Worksheet 2.14. Framework for Summarizing
Information about Performance Measures

Activity 4. Report Results to State

PROTOCOL TWO | 96

Worksheet Name

Protocol Activity and Step

Worksheet 2.15. Performance Measure Validation
Reporting Template

Activity 4. Report Results to State

Worksheet 2.16 Performance Measure Reporting
Template

Activity 4. Report Results to State

PROTOCOL TWO | 97

Worksheet 2.1. List of Performance Measures to be Validated
Instructions. In the table below, identify the performance measures to be validated, the data source, reporting
frequency, and format as described in Activity 1. Step 1. Complete the worksheet for each measure to be validated,
adapting as needed.
Acronyms: CMIT = CMS Measures Inventory Tool; MRR = Medical Record Review; NCQA = National Committee for
Quality Assurance.

Performance
measures

CMIT #

98 | PROTOCOL TWO

Admin.
data

MRR

Hybrid
(admin.
data and
MRR)

Electronic
Health
Record

Survey

Reporting
frequency
and format

Comments

Worksheet 2.2. Performance Measure Validation Template
Instructions. For each performance measure, use this template to gather audit specifications for the validation components (as described in Activity 1. Steps 1
and 3, and Activity 2. Step 4) and to assess the MCP’s measurement and reporting process for each component.
For each validation component, indicate whether the measure meets validation requirements by checking “Yes,” “No,” or “Not Applicable.” Insert comments to
explain “No” and “Not Applicable” responses. Use the following guidance to assess each component.
•

Yes: The MCP’s measurement and reporting process was fully compliant with state specifications

•

No: The MCP’s measurement and reporting process was not fully compliant with state specifications. This designation should be used for any validation
component that deviates from the state specifications, regardless of the impact of the deviation on the final rate. All components with this designation
must include an explanation of the deviation in the comments section

•

NA: The validation component was not applicable. Include an explanation in the comments section (e.g., sampling not required, medical record review
not included)

Acronyms: CPT = Current Procedural Terminology; DRG= Diagnosis Related Group; EHR = Electronic Health Record; ICD = International Classification of
Diseases; ID = Identification Number; LOINC = Logical Observation Identifiers, Names, and Codes; MCP= Managed Care Plan; MRR = Medical Record Review;
NA = Not Applicable; NR = Not Reported.
MCP name: __________________________________________________________________
Performance measure: __________________________________________________________
Method for calculating measure: [ ] Admin

[ ] MRR [ ] Hybrid [ ] EHR [ ] Survey

Validation Findings
Validation component

Audit
specifications

Yes

Comments

Documentation:
Did appropriate and complete measurement plans and
programming specifications exist, including data sources,
programming logic, and computer source code?
Were internally developed codes used?

PROTOCOL TWO | 99

Validation component

Audit
specifications

Denominator:
Were all the data sources used to calculate the
denominator complete and accurate (e.g., eligibility files,
c