eCQI Tools & Key Resources

The standards and tools versions listed for each reporting/performance period are the versions used to create and/or support the implementation of the specific reporting/performance period specifications. Newer versions of the standards and tools may be available, but were not used in the development of the reporting/performance period listed.

The eCQM tools and resources provide a foundation for the development, testing, certification, publication, implementation, reporting, and continuous evaluation of quality measures and their improvement. You can refine the tool list by selecting a category of interest and/or a role that best describes your needs, or you can also click a specific tool from the list below to view additional details.

Binding Parameter Specification (BPS)

Category: Implementation

The value set Binding Parameter Specification (BPS) is generated by the National Libray of Medicine(NLM).  It is a record of the value set metadata information that defines the value set code lists specified by published CMS eCQMs. Measure implementers and vendors can use the BPS to track versions and other parameters that define the value set code lists for each eCQM release. The BPS contains the same information as its predecessor, the discontinued Data Element Catalog (DEC), as well as additional parameters. Users can access the BPS metadata file, for any released set of CMS eCQM value set files, through the NLM's Value Set Authority Center Downloadable Resources.

Bonnie

Category: Development, Testing
Who Uses Tool: Measure Developer

Software tool that allows eCQM developers to test and verify the behavior of their eCQM logic. The main goal of the Bonnie application is to reduce the number of defects in eCQMs by providing a robust and automated testing framework. The Bonnie application allows measure developers to independently load measures that they have constructed using the MAT. Developers can then use the measure metadata to build a synthetic patient test deck for each measure from the clinical elements defined during the measure development process. By using measure metadata as a basis for building synthetic patients, developers can quickly and efficiently create a test deck for a measure. The Bonnie application helps measure developers execute the measure logic against the constructed patient test deck and evaluate whether the logic aligns with the intent of the measure.

Bonnie User Group

Category: Development, Testing
Who Uses Tool: Measure Developer

Community of Bonnie users that benefit from one another’s experience and questions, and allows the development team to easily reach out to the community regarding new releases, proposed features, and focus group opportunities.  Follow these instructions to join:

  • Navigate to https://groups.google.com/forum/#!forum/bonnie
  • Sign into a google account, or create one
    • You can create a google account tied to an existing email address: under “Choose your username” click “I prefer to use my current email address.”
  • Click “Join group”
  • Adjust your preferred settings in the pop-up dialog to indicate how you wish to receive group messages. You can change these settings later
  • Click “Join this group”

CDS Connect

Category: Continuous Evaluation/Feedback, Development, Publication

Web-based repository service which enables the Clinical Decision Support (CDS) community to identify evidence-based standards of care, translate and codify information into an interoperable standard, and leverage tooling to promote a collaborative model of CDS development. 

Cedar

Category: Development, Testing

Open source tool for testing the strength of electronic Clinical Quality Measure (eCQM) collection systems that receive Quality Reporting Document Architecture (QRDA) files.

Certified Health IT Product List (CHPL)

Category: Certification

Comprehensive and authoritative listing of all certified Health Information Technology which have been certified by the ONC Health IT Certification program. All products listed on the CHPL have been tested by an ONC-Authorized Testing Labs (ONC-ATLs) and certified by an ONC-Authorized Certification Body (ONC-ACB) to meet criteria adopted by the Secretary of the Department of Health and Human Services (HHS).

For additional information on how to navigate the CHPL, please refer to the CHPL Public User Guide.

Clinical Quality Framework

Category: Continuous Evaluation/Feedback

Collaborative community of participants from the public and private sectors focused on providing tools, services and guidance to facilitate the harmonization of standards for clinical decision support (CDS) and electronic clinical quality measures (eCQM) use in clinical quality improvement.

CMS Measures Inventory Tool (CMIT)

Category: Publication

Repository of record for information about CMS measures used to promote healthcare quality and quality improvement. CMS and its partners use this measure inventory to inform stakeholders, to manage its measure portfolio, and to guide measure development. The inventory contains information describing each of the measures including title, numerator, denominator, exclusions, various identifiers, type, status, usage by program, steward, healthcare priorities, and other attributes. The intuitive and user-friendly functions allow you to find measures quickly and to compile and refine sets of related measures. The tool increases transparency, can be used to identify measures across the continuum of care, and will help coordinate measurement efforts across all conditions, settings, and populations.

CMS QRDA Pre-Submission Validation Tools Guide

Category: Reporting

Resource which details the different aspects of multiple QRDA validation tools that are available to submitters. It is intended to give users a single point of reference for these tools and assist them in selecting the most appropriate tool to meet their individual needs. The tool covers:

  • Cypress QRDA Validation Utility
  • Pre-Submission Validation Application (PSVA)
  • CMS Data Receiving System, within the QualityNet Secure Portal

CMS Quality Reporting Document Architecture (QRDA) Implementation Guides (IGs)

Category: Implementation, Reporting

These IGs are based on the HL7 QRDA IGs. CMS's QRDA Category I IG is used for Hospital Quality Reporting (HQR), and CMS's QRDA Category III IG is used for the Eligible Clinicians and Eligible Professionals Programs that address aggregate reporting requirements for Comprehensive Primary Care Plus (CPC+) and Merit-Based Incentive Payment System (MIPS) programs. HL7 Quality Reporting Document Architecture (QRDA) Implementation Guides (IGs) These IGs describe constrains on HL7's Clinical Document Architecture Release 2 (CDA R2) to create QRDA documents. QRDA Category I is for reporting quality data at the individual patient level. QRDA Category III is for reporting aggregated quality measure data

CMS Receiving System Edits Document

Category: Reporting

Resource which provides greater insight into the conformance error messages and system edits for the 2017 CMS Receiving System. Details outlined in the resource include the specific conformance error ID, the associated error message, the template ID or template name where the error occurred, and the condition or content expected within the QRDA I file to address the conformance error.

CQL Online

Category: Development, Implementation

CQl tools which provide a "sandbox" where electronic Clinical Quality Measure (eCQMs) developers and implementers can "play" with the Clinical Quality Language (CQL). The tools allows users to get a feel for what CQL looks like and how to write proper expressions in this new and powerful language.

More specifically, CQL Online is a syntax directed editor for writing expressions in CQL. It is open source and intended for educational purposes.

CQL Style Guide

Category: Development

CQL Style Guide provides examples to standardize expression of measure concepts across eCQMs and define a uniform “look and feel” to eCQM logic using CQL. The guide focuses on a set of common best practices implemented across CQL-based eCQMs in CMS reporting programs and also promotes the use of consistent language within the framework of CQL, including libraries, aliases, definitions, functions and conventions.

CQL-to-ELM Translator Reference Implementation

Category: Implementation

The HL7 CQL specification defines both a high-level, author friendly syntax for expressing clinical knowledge, as well as a machine friendly, syntax-independent canonical representation of clinical knowledge called Expression Logical Model (ELM). The high level syntax is designed to support measure and decision support authors, while the canonical representation is designed to support point-to-point sharing and machine processing applications.

The specification describes a formal mechanism for translating the high-level CQL syntax into the canonical ELM representation. This project provides a reference implementation of that translation. The translation component is designed to support use within an Integrated Development Environment (IDE), and has several options and features focused on providing translation support in such an environment. The reference implementation is intended to be used in support of CQF implementations as a tool to enable CQL output to be uniformly and automatically translated into ELM XML or JSON documents for sharing and distribution to support implementation, integration, translation, and execution of CQL-based artifacts.

The CQL-to-ELM Translator is substantially complete, supporting all language constructs of the latest version of CQL (Release 1 STU 2 (1.2)), as well as a broad range of functionality to support use of the translator in an Integrated Development Environment. Implementations making use of the translator can submit issues and track resolution progress through the Issues tracker in the Github repository.

Cypress

Category: Certification, Testing

Open source testing tool used by vendors to certify their EHRs and health information technology (IT) modules for calculating eCQMs. Cypress is an official testing tool for the ONC EHR Certification Program. Testing involves Cypress generating synthetic patient records for the subset of published eCQMs selected for certification and testing the ability of the EHR systems and health IT modules to accurately record, import, calculate, filter, and report eCQMs. The Cypress test data are available in QRDA Category I for import into an EHR system. Cypress tests an EHR system’s ability to generate accurate QRDA Category I and Category III documents for reporting to CMS. Learn more about the Cypress Tool by visiting the Cypress website and reading the Cypress User Guide.

Cypress Tech Talks

Category: Continuous Evaluation/Feedback

Open forum for discussing technical implementation and tool issues with the Cypress support team.

Cypress Validation Utility

Category: Certification, Testing

Conformance validation tool which provides implementers with the ability to validate the conformance of QRDA Category I and Category III documents to the 2016, 2017, and 2018 CMS implementation guides:

  • 2016 (EP/EH)
  • 2017 (EP/EH)
  • 2018 (EH Only)

and HL7 implementation guides:

  • QRDA Category I (R3, R3.1, R4)
  • QRDA Category III (R1, R2.1)

eCQI Implementers' Corner

Category: Implementation

Webpage designed to support the eCQI implementation activities of the entire care team.  The page provides links to resources and past presentations on implementing eCQMs.

eCQI Resource Center

Category: Publication

Website which provides eCQI resources and connections with the community of professionals who are dedicated to electronic clinical quality improvement for better health. It provides eCQM electronic specifications along with tool and resource links. It serves as “the one-stop shop for the most current resources to support electronic clinical quality improvement.”

eCQM Calculations Libraries and Technical Services

Category: Continuous Evaluation/Feedback, Testing

Open source libraries which provide a proof-of-concept implementation of eCQMs against patient information represented in Fast Healthcare Interoperability Resources (FHIR). These libraries reuse the calculation engine developed within Bonnie and Cypress and adapt it to a FHIR data model. They can be used as a starting point for an implementation of eCQMs from FHIR-based data.

  
fhir-patient-api maps the current Quality Data Model (QDM) based eCQMs to FHIR resources.

node-qme a “patient source”, which is responsible for using the fhir-patient-api to retrieve patients from a database and send them to eCQM code for calculation.

node-cqm-engine  monitors requests for eCQM calculations and executes them. It also aggregates the results from individual patient measure results into a result for the entire patient population.

ecqm server  a server application that provides RESTful interfaces for eCQM calculation.

ecqm-frontend  a software package that provides a web-based interface for users to create and view the results of eCQM calculations.

Federal Register

Category: Publication

Official daily publication for rules, proposed rules, and notices of federal agencies and organizations, as well as executive orders and other presidential documents.  The Federal Register is published by the Office of the Federal Register, National Archives and Records Administration (NARA). 

HL7 Clinical Quality Language (CQL)-based Health Quality Measure Format (HQMF) Implementation Guide (IG)

Category: Development

The CQL-based HQMF IG constrains the HL7 HQMF IG by utilizing both a domain analysis model called the QDM and an expression logic called CQL. QDM allows quality measure developers to describe clearly and unambiguously the data required to calculate quality measures. QDM data types are mapped to the HQMF templates. CQL allows the sharing of unambiguous logic among eCQMs and can be used with multiple data models.

HL7 Quality Data Model (QDM)-based Health Quality Measure Format (HQMF) Implementation Guide (IG)

Category: Implementation

The QDM-based HQMF IG constrains the HL7 HQMF IG by utilizing a domain analysis model called the QDM. This allows quality measure developers to describe clearly and unambiguously the data required to calculate quality measures. QDM data types are mapped to the HQMF templates. No stand-alone QDM-based HQMF IG
has been created since October 2016 and its functionality has been superseded by the CQL-based HQMF IG.

HL7 Quality Reporting Document Architecture (QRDA) Category I Implementation Guide (IG)

Category: Implementation

The HL7 QRDA I IG describes constrains on HL7's Clinical Document Architecture Release 2 (CDA R2) to create a QRDA I document. QRDA Category I is for reporting quality data at the individual patient level.

HL7 Quality Reporting Document Architecture (QRDA) Category III Implementation Guide (IG)

Category: Implementation

The HL7 QRDA III IG describes constrains on HL7's Clinical Document Architecture Release 2 (CDA R2) to create a QRDA III document. QRDA Category III is for reporting aggregated quality measure data.

Kaizen

Category: Continuous Evaluation/Feedback

Kaizen, or continuous improvement, is a long-term approach to work that systematically seeks to achieve incremental changes in order to improve efficiency and quality. It focuses on removing process waste and maximizing value to the customer (in this case the patient). CMS and ONC have focused on eCQM development and implementation.

Measure Authoring Tool (MAT)

Category: Development

Web-based tool that allows measure developers to author eCQMs using the Quality Data Model (QDM). The tool provides the capability to express measure logic and export measures. The MAT provides the capability to express complex measure logic and export measures in several formats, including a human-readable document which can be viewed in a web browser (HTML), a CQL file containing the CQL library in its entirety, and for integration into Electronic Health Records (EHRs), the MAT provides an eCQM HQMF XML document, an Expression Logical Model (ELM) XML document, and a corresponding JavaScript Object Notation (JSON) file. Measure developers use both Bonnie and MAT to promote test driven development. Refer to the MAT User Guide for more information.

Measure Logic Guidance Document

Category: Implementation

Resource document which provides guidance for understanding, using, and/or implementing eCQMs.

Measures Management System

Category: Continuous Evaluation/Feedback, Development

Standardized system for developing and maintaining the quality measures used in CMS’s various quality initiatives and programs. The primary goal of the MMS is to provide guidance to measure developers to help them produce high-caliber healthcare quality measures.

Measures Management System Blueprint

Category: Continuous Evaluation/Feedback, Development, Implementation, Testing

Guidance document which provides a standardized system for developing and maintaining the quality measures used in various quality initiatives and programs. Includes information on Measure Development Concepts, Measure Lifecycle, eCQM Lifecycle, as well as Tools, Appendices, and Forms.

Measures Management System Measure Inventory

Category: Development, Implementation, Publication

Inventory of measures used by CMS in various quality, reporting, and payment programs. The Inventory lists each measure by program and contains descriptive information including, but not limited to, numerator, denominator, exclusion criteria, National Quality Strategy (NQS) domain, measure type, and National Quality Forum (NQF) endorsement status. 

The CMS Measures Inventory now includes Pipeline/Measures under Development (MUD) in a separate document, which are measures that are in progress, but are not fully developed. The Measures under Development are categorized by anticipated CMS program, condition/topic, measure type, NQF endorsement status, measure steward, and measure contractor.

NCQA eCQM Crosswalk

Category: Certification, Testing

Crosswalk which identifies eCQMs that are tested under the NCQA eMeasure (eCQM) Certification program, accepted in the NCQA PCMH Recognition Program, as well as those selected as core America’s Health Insurance Plans (AHIP)/CMS eCQMs and used within NCQA’s Healthcare Effectiveness Data and Information Set (HEDIS®) reporting. 

The same eCQMs that are used for CMS Quality Payment Program (QPP) reporting can also be submitted for recognition under NCQA’s Patient-Centered Medical Home (PCMH) recognition programs.

NCQA’s eMeasure (eCQM) Certification Program

Category: Certification, Testing

eCQM certifcation program which uses ONC-approved methodology for testing Health IT's ability to create accurate eCQMs.  The NCQA eCQM Certification program is for use in the ONC HIT Certification Program and is approved as an Authorized Test Lab (ATL).

NQF Quality Positioning System (QPS)

Category: Implementation

Web-based inventory tool developed by the National Quality Forum (NQF) to help people select and use NQF-endorsed measures. It allows a user to search for NQF-endorsed measures in a number of ways: export search results; and view types of measures. Using QPS, a user can find NQF-endorsed measures on particular topics, track and get reminders about measures that are important to them, provide feedback on measures, and see which measures others are using.

NQF-endorsed measures have undergone careful review by expert multistakeholder committees made up of doctors, hospitals and other providers, employers, health plans, public agencies, community coalitions, and patients –most of whom use measures on a daily basis to ensure better care. QPS is designed to help anyone interested in quality measurement. Information in QPS is updated daily as measures move through NQF processes. By adding measures to a portfolio in QPS, users can receive email notification of changes to individual measures.

Submission guidance for eMeasures is at: www.qualityforum.org/Electronic_Quality_Measures.aspx

ONC Project Tracking System

Category: Certification, Continuous Evaluation/Feedback, Implementation, Testing

Collaboration platform hosted by the HHS’s Office of National Coordinator for Health Information Technology (ONC) which supports the implementation of health information technology. It provides internal and external users with a common place to transparently log, prioritize, and discuss issues with appropriate subject matter experts on many topics. CMS uses JIRA during most phases of the eCQM measure lifecycle, including development, implementation, and maintenance. JIRA supports projects such as CMS’s annual Measures under Consideration process and ONC’s health IT certification program.

Key topic areas covered include the eCQM Issue Tracker (eCQM implementation), QDM Issue Tracker (QDM measure logic), QRDA Issue Tracker (QRDA implementation), Bonnie Issue Tracker (eCQM testing), and USHIK feedback. Policy questions are not covered in JIRA.

Pioneers in Quality

Category: Implementation

A Joint Commission program to assist hospitals on their journey towards electronic clinical quality measure adoption  including educational programs (e.g., webinars for CEUs), a resource portal, recognition categories, an advisory council, a modified annual report, speaker’s bureau outreach, a peer–to–peer solution exchange, as well as having a strong focus on partnering with hospitals to provide the highest level of quality care for patients and their families.

popHealth

Category: Reporting

Open source software tool that can automate the reporting of summary quality measures and streamline the exchange of summary quality data. popHealth can be used by developers to better understand quality measurement and reporting. The tool has also proven valuable for its individual parts: standardized data importer/exporter; user interface; and most importantly its quality measure engine, which is technology shared with Cypress. EHR and HIE developers have utilized these components in different ways to improve their own products.

Pre-Submission Validation Application (PSVA)

Category: Reporting, Testing

Application Program Interface (API) which provides vendors, hospitals and providers with a method for validating eCQM files within their own system/environment prior to submission and for securely transporting valid files to Centers for Medicare & Medicaid Services (CMS). The PSVA tool assists in identifying and correcting Quality Reporting Document Architecture (QRDA) Category I file format errors for test and production files prior to data submission to the CMS eCQM data receiving system QualityNet Secure Portal. The PSVA tool has three interfaces as part of the download package and a user is able to select the option that best meets their needs. The PSVA tool can be accessed on the QualityNet Secure Portal.

QRDA Category I Conformance Statement Resource

Category: Reporting, Testing

Resource which assists data submitters in resolving file upload errors by providing additional explanations and reference material to support successful submission.

As of Calendar Year 2017, Quality Reporting Document Architecture (QRDA) Category I files are submitted by Eligible Hospitals (EHs) to fulfill a portion of the mandatory electronic Clinical Quality Measure (eCQM) reporting requirements for the Centers for Medicare & Medicaid Services (CMS) Hospital Inpatient Quality Reporting (IQR) and Medicare Electronic Health Record (EHR) Incentive Programs.

QRDA III Converter

Category: Reporting, Testing

Converter tool which converts QRDA III files to the QPP JSON (JavaScript Object Notation) format for Quality Payment Program (QPP) data submission. The tool can also be used to test whether QRDA III files can be consumed by QPP and allows you to see whether it meets these standards:

This tool does not have the same functionality as the Cypress tool and cannot be used to test QRDA III files against the full range of possible uses. For more information, visit the Quality Payment Program website.

Designed for IT developers and available in the CMS Github repository.

QRDA-CQM Sandbox

Category: Implementation

Cypress testing environment which contains resources and tools for the QRDA Category I and Category III standards. The QRDA Validator provides implementers with the ability to validate the conformance of HL7 QRDA Category I, R2 and QRDA Category III, R3 standards. Note that CMS programs may have additional program specific requirements for QRDA submissions to these programs, which are specified in the CMS QRDA Implementation Guide. 

Quality Data Model (QDM)

Category: Development

Information model that defines relationships between patients and clinical concepts in a standardized format to enable electronic quality performance measurement. The model is the current structure for electronically representing quality measure concepts for stakeholders involved in electronic quality measurement development and reporting. The QDM provides the language that defines the criteria for clinical quality measurement. It allows the electronic definition of a clinical concept via its data elements and provides the vocabulary to relate them to each other. By relating attributes between data elements and using filtering functions, the QDM provides a method to construct complex clinical representations for eCQMs.

Quality Net

Category: Reporting

Website which provides healthcare quality improvement news, resources and data reporting tools and applications. QualityNet is the only CMS-approved site for secure communications and healthcare quality data exchange between healthcare organizations and facilities.

Quality Payment Program (QPP) Developer Tools

Category: Reporting

Developer tools for the Quality Payment Program allow industry developers who build software for participating clinicians to integrate their software directly with CMS applications and data. 

The following tools are available:

These include documentation for reporting eCQMs used in the QPP program through QRDA-III, QPP JSON or QPP XML formats.

Release Notes - eCQM Eligible Hospital

Category: Implementation

Resource file which provide an overview of technical changes in the eCQM Annual Update and addenda.

Release Notes - eCQM Eligible Professional/Clinician

Category: Implementation

Resource files which provide an overview of technical changes in the eCQM Annual Update and addenda.

The United States Health Information Knowledgebase (USHIK)

Category: Development, Implementation, Publication

Registry and repository of health care-related metadata, specifications, and standards. The CMS Quality Programs portal provides the ability to view, download, and compare eCQM specifications, data elements, value sets, and codes released by CMS and NLM. Each eCQM release package is available in a single file, or can be downloaded as eligible hospital, eligible professional, and individual measure packages. These files are available in PDF, Excel, CSV, HQMF XML, or XML. UHSIK provides the ability for public comments to be received on measures and measure updates under consideration. This knowledgebase is produced by the Agency for Healthcare Research and Quality (AHRQ) in partnership with CMS and NLM.

Note: Viewing and/or downloading proprietary value sets, such as those using AMA copyrighted CPT codes or SNOMED CT codes, from NLM-VSAC or AHRQ-USHIK requires a free Unified Medical Language System® Metathesaurus License (UMLS), (available at https://uts.nlm.nih.gov/license.html). It is expected that any use of value sets is consistent with these licensing requirements and copyright protections.

Unified Medical Language System® (UMLS®)

Category: Development

Set of files and software that bring together health and biomedical vocabularies and standards to enable interoperability between computer systems. UMLS® is used to enhance or develop applications, such as electronic health records, classification tools, dictionaries and language translators.  Use of UMLS allows end users to view codes used in value sets such as CPT and ICD-10.

Value Set Authority Center (VSAC)

Category: Development, Implementation, Publication, Testing

Central repository for the official versions of value sets that support the electronic Clinical Quality Measures. The National Library of Medicine (NLM) maintains the NLM Value Set Authority Center and provides downloadable access to the value sets and the Binding Parameter Specification. The VSAC provides measure developers with tools to search existing value sets, collaborate with other measure developers to harmonize value sets, to create new value sets, and to maintain value set content consistent with current versions of the terminologies they use.

Value sets are lists of specific values (terms and their codes) used to describe clinical and administrative concepts in the quality measures. They provide groupings of unique values along with a standard description or definition from one or more standard vocabularies used to describe the same clinical concept (e.g., diabetes, clinical visit, demographics) within the quality measures.

Information on how value sets are created, used, and updated is available in the VSAC Support Center.

Note: Viewing and/or downloading proprietary value sets, such as those using AMA copyrighted CPT codes or SNOMED CT codes, from NLM-VSAC or AHRQ-USHIK requires a free Unified Medical Language System® Metathesaurus License (UMLS), (available at https://uts.nlm.nih.gov/license.html). It is expected that any use of value sets is consistent with these licensing requirements and copyright protections.

VSAC Collaboration Tool

Category: Development

Tool which provides a central site where value set authors can post value sets for collaborative discussion. In that site, teams can share threaded discussions about the value sets, view recent value set expansions posted by site members, organize their value sets by usage and by team’s workflow needs, and receive activity and change notifications from VSAC. VSAC authors and stewards can access the VSAC Collaboration Tool from within VSAC’s Collaboration Management tab as well as through a direct link to the VSAC Collaboration Tool.