Glossary

Agile development is a term used to describe iterative software development used to shorten the software development lifecycle and emphasizes flexibility, collaboration, and customer satisfaction. Agile development teams execute the entire software development lifecycle in smaller increments, usually called sprints, typically 1-4 weeks long. The software development industry often contrasts agile development with traditional or waterfall development, where the planning for larger projects occurs up front and executed against that plan. Agile development is an iterative approach to development with regular feedback loops or intervals. These iterations allow diversion of a team to be productive in one part of a project while resolving a problem or issue in another part. 

Alignment with respect to the process of ensuring that similar standardized quality measures are used consistently across different programs and sectors, both in government and private initiatives. This involves harmonizing measures so that they are compatible and comparable, which helps in providing clear and consistent information for patients, consumers, and stakeholders. Alignment is achieved when a set of measures works well across settings or programs to produce meaningful information without creating extra work for those responsible for the measurement.

An analytic model for developing clinical practice guidelines is a visual representation of a causal pathway showing the linkages between how the proposed key research question(s) and intervention(s) under consideration as reflected in the PICOTS (P-Patient population; I- Intervention; C- Comparator; O- Outcome; T- Timing; S- Setting) framework and to their intended outcomes requiring confirmation by evidence review to support related recommendations. The linkages in the model represent critical logical premises and presumed relationships among intermediate, surrogate, and ultimate health outcomes related to a specified clinical question. Woolf, S., Schünemann, H. J., Eccles, M. P., Grimshaw, J. M., Shekelle, P. (2012). Developing clinical practice guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations. Implementation Science, 7(61). https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-7-61

An application programming interface (API) is a system of tools and resources in an operating system enabling developers to create software applications. An API is a software intermediary allowing two applications to talk to each other. Each time you use an app like Facebook, send an instant message, or check the weather on your phone, you are using an API. Frye, M-K. (n.d.). What is an API? Retrieved September 13, 2023, from https://www.mulesoft.com/resources/api/what-is-an-api

A computational artifact is anything created by a human using a computer. An artifact can be, but not limited to a code, program, image, audio, video, presentation, or web page file. 

The ASTP/ONC Project Tracking System (Jira) is a collaboration platform hosted by the Assistant Secretary for Technology Policy (ASTP)/Office of National Coordinator for Health Information Technology (ONC). It provides users with a common place to transparently log, track, and discuss issues with subject matter experts on many topics, including electronic clinical quality measure (eCQM) implementation. CMS uses Jira during most phases of the eCQM lifecycle, including development, implementation, and maintenance.  The ASTP/ONC Project Tracking  System (Jira) and Confluence have a 90-day account deactivation policy for all users who have not logged into systems for a period of 90 days or longer. 

Reminder: 
This is an open platform that does not intend for users to communicate sensitive or confidential information such as protected health information and personal identifiable information.

The value set Binding Parameter Specification (BPS) is generated by the National Library of Medicine (NLM) as part of the Value Set Authority Center’s (VSAC) Downloadable Resources. The BPS details the usage of value sets by electronic clinical quality measures (eCQMs), the binding of value sets to QDM categories, and the value sets object identifiers (OIDs) used to create grouping value sets. It also lists other select value set metadata, allowing eCQM implementers and vendors to use the BPS to track versions and other parameters of the value sets for each eCQM release. Users can access the BPS metadata file for any released set of CMS eCQM value set files through the NLM's VSAC Downloadable Resources. Access requires a free Unified Medical Language System (UMLS) license.

A case presentation is a formal communication between health care professionals (e.g., physicians, pharmacists, nurses, therapists, nutritionists) providing a synthesis of a patient's clinical information. Components of a case presentation typically include identifying information, reason for consultation/admission, chief complaint (in the patient's own words), recent history, active medical problems, medications, allergies, social history, physical examination findings, relevant test results, assessment, and treatment plans.

Clinical Decision Support (CDS) Connect is a freely available web-based platform, including an authoring tool and a repository, enabling the CDS community to identify evidence-based care, translate and codify information into an interoperable health information technology standard, and leverage tooling to promote a collaborative model of CDS development.

Clinical Decision Support (CDS) Hooks is a Health Level Seven International® (HL7®) specification managed by the HL7 Clinical Decision Support (CDS) Workgroup providing a way to embed additional, near real-time functionality within a clinician’s workflow of an electronic health record. A specified event(s) within the clinician’s workflow triggers the CDS service, which gathers the appropriate specified data elements through secure Fast Healthcare Interoperability Resources® (FHIR®) services. By utilizing FHIR services, CDS Hooks provides interoperability between multiple stakeholders operating on different platforms.

The FHIR service app can return appropriate guidance, such as proposed alternative medication or formulary information. CDS Service vendors can test CDS Service functionality using the CDS Hooks Sandbox. More advanced functionality is under development, such as the multi-interested party Da Vinci Burden Reduction Project and provides use cases for alerts/notifications, prior authorization, and document exchange. The Agency for Healthcare Research and Quality's CDS Connect Project provides a web-based platform to assist with development of CDS artifacts and CDS Hooks.

The Certified Health Information Technology (IT) Product List (CHPL) is a comprehensive and authoritative listing of all certified health information technology that have been successfully tested and certified by the ONC Health IT Certification Program. All products listed on the CHPL have been tested by an ONC-Authorized Testing Laboratory (ONC-ATL) and certified by an ONC-Authorized Certification Body (ONC-ACB) to meet criteria adopted by the Secretary of the Department of Health and Human Services (HHS). For additional information on how to navigate the CHPL, please refer to the CHPL Public User Guide.

ClinFHIR is an open-source tool providing an educational environment and also allows health information technology developers to create or search for Fast Healthcare Interoperability Resources® (FHIR®)-based resources. It serves as a training tool to help people wanting to learn more about FHIR visualize how the parts combine to represent clinical information in a structured and coded manner. It also serves as a development tool with features to build some of the required artifacts, particularly as an aid to learning.

Clinical decision support is health information technology functionality building upon the foundation of an electronic health record to provide persons involved in care processes with general and person-specific information, intelligently filtered and organized, at appropriate times, to enhance health and health care.

Clinical decision support alert fatigue occurs when a clinician, after receiving too many alerts or reminders, begins to override or ignore further alerts without attending to them, thus potentially decreasing the care improvements expected from the tools.

The most commonly implemented clinical decision support (CDS) alerts prompt clinicians about guidance, e.g., drug-allergy, drug-drug, and drug-disease warnings, or provide dosing guidance. Passive CDS includes order sets, patient data reports, and documentation templates while active CDS includes rules and alerts usually delivered through computerized provider order entry or other functions of the electronic health records.

Clinical decision support (CDS) artifacts are items representing medical knowledge from various knowledge sources (e.g., clinical guidelines, peer-reviewed articles, local best practices, and clinical quality measures). The artifacts can take many forms, but the goal is to create computable, interoperable translations using Clinical Quality Language. CDS artifacts are found in the Repository. A variety of organizations, including federal agencies, contribute CDS artifacts to the Repository and they span a number of topics including anatomy, health care, and diseases.

A clinical decision support (CDS) developer is an individual or organization that translates knowledge to a structured and/or executable tool aiding in making evidence-informed decisions about a patient’s health care. CDS developers may or may not be the original knowledge authors (e.g., guideline developers, subject matter experts) or the final implementers. They ensure accurate and consistent reflection of the original clinical knowledge in the appropriate standard coding schemes (e.g., Clinical Quality Language and terminologies such as Current Procedural Terminology and SNOMED CT), accounting appropriately for intellectual property and licensing.

Clinical decision support (CDS) implementation activities refer to configuration, customization, and other needed steps for health information technologies to function for a specific organization or group of end-users. For example, with CDS, the local implementation will consider end-user workflows to identify the points at which to present the CDS, determine which data fields in local databases will map to data fields that drive the CDS, and educate end-users about important features of the CDS. Typically, this involves translation of guideline recommendations from L3 to L4 or other local adaptation (localization) (See Levels of knowledge representation for CPG).

Clinical Document Architecture (CDA) is a popular, flexible markup standard developed by Health Level Seven International® defining the structure of certain patient medical records, such as discharge summaries and progress notes, as a way to better exchange this information between health care providers and patients. Wallask, S. (n.d.). Clinical document architecture (CDA). TechTarget: Health IT. Retrieved March 12, 2024, from https://searchhealthit.techtarget.com/definition/Clinical-Document-Architecture-CDA

Clinical information systems "are computer systems that provide immediate access to current patient data regarding clinical notes, medication history, laboratory reports, images, and reports either directly or via data networks. They are parts of a hospital information system, which facilitates direct patient care." Islam, M. M., Poly, T. N., & Li, Y-C. J. (2018). Recent advancement of clinical information systems: Opportunities and challenges. Yearbook of Medical Informatics, 27(1), 83–90. https://www.thieme-connect.com/products/ejournals/abstract/10.1055/s-0038-1667075

Clinical practice guidelines (CPGs) are systematically developed statements to assist clinician and patient decisions about appropriate health care for specific clinical circumstances. CPGs are statements that include recommendations intended to optimize patient care. They are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options. Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. (2011). Clinical practice guidelines we can trust. Institute of Medicine. https://www.ncbi.nlm.nih.gov/books/NBK209539/pdf/Bookshelf_NBK209539.pdf  

A clinical quality measure (CQM) is a mechanism used for assessing the degree to which a clinician competently and safely delivers clinical services appropriate for the patient in an optimal time frame. CQMs are a subset of the broader category of quality measures.

CMS Measures Inventory Tool (CMIT) is the repository of record for information about CMS quality measures. CMS and its partners use the inventory to inform stakeholders, manage its measure portfolio, and guide measure development. The functions allow users to find measures quickly, compile and refine sets of related measures, identify measures across the continuum of care, and help coordinate measurement efforts across all conditions, settings, and populations.

The CMS consensus-based entity (CBE) assigns the CMS CBE identification number to a measure that has successfully gone through the CBE endorsement process. The CMS CBE Submission Tool and Repository (STAR) is the database of record of CBE-endorsed measures. 

The Medicare Improvements for Patients and Providers Act of 2008 requires the U.S. Department of Health and Human Services to contract with a consensus-based entity (CBE) regarding performance measurement. The CMS CBE endorses quality measures through a transparent, consensus-based process incorporating feedback from diverse groups of interested parties to foster health care quality improvement.

The CBE also convenes multi-interested party groups to discuss the Measures Under Consideration List as part of the pre-rulemaking process and reviews CMS's existing measure portfolio for possible change recommendations.

The CMS Data Element Library (DEL) is the centralized resource for CMS assessment instrument data elements (e.g. questions and responses) and their associated health information technology standards. It currently includes CMS's post acute care assessment instruments.

When a measure developer creates a new electronic clinical quality measure in the Measure Authoring Development Integrated Environment (MADiE), MADiE assigns the CMS eCQM identification to the new measure. The CMS eCQM ID is an essential data element when submitting eCQM data to CMS.

A CMS Electronic Health Record (EHR) Certification ID is a number generated by the Certified Health Information Technology (IT) Product List (CHPL) and used for reporting to CMS. It represents a single product or combination of products in the CHPL. The CMS EHR Certification ID is different from the CHPL product number. In the CHPL, this is the number generated when the hospital/clinician selects EHR Certification ID for a suite of products making up the hospital's/clinician's EHR solution. If a product changes, then a different CMS EHR Certification ID will be generated. The CMS EHR Certification ID is only unique to the product suite. If two different hospitals/clinicians happen to use the same products, then they will both have the same CMS EHR Certification ID.

CMS requires the CMS EHR Certification ID in several of its quality reporting programs. Check with individual quality reporting programs for requirements.

The CMS Measures Under Consideration Entry/Review Information Tool (CMS MERIT) is the web-based data collection portal allowing measure developers to submit candidate measures to CMS for consideration. The Tool walks the submitter through the process of providing information. Each submitter has a dashboard allowing them to track the progress of their submitted measure and to communicate with CMS through the comment function. The Tool also provides a view of all measures submitted for the current year. Another purpose of CMS MERIT is to give federal officials, the consensus-based entity multi-stakeholder groups, and other stakeholders more information as they review measures.

The CMS Measures Management System (MMS) Hub is a resource providing information about the MMS, guidance for measure developers (content previously found in the Blueprint), and quality measure-related news and events. It includes information on measure development, measure implementation, measure maintenance, tools, templates, opportunities to participate in public comment and technical expert panels, and more.

The CMS National Quality Strategy (NQS) is a long-term initiative with the aim to promote the highest quality outcomes and safest care for all individuals. The CMS NQS has four priority areas: Outcomes and Alignment, Equity and Engagement, Safety and Resiliency, and Interoperability and Scientific Advancement. 

The CMS Quality Reporting Document Architecture (QRDA) Implementation Guides (IGs) are based on the Health Level Seven International® QRDA IGs. The CMS QRDA Category I IG is used for hospital CMS quality reporting programs. 

The CMS Quality Reporting Document Architecture (QRDA) Pre-Submission Validation Tools are resources offering QRDA validation tools available to submitters. The intent is to give users a single point of reference for these tools and assist them in selecting the most appropriate tool to meet their individual needs. The tools cover

• Cypress
• Hospital Quality Reporting System via Healthcare Quality Information System (HCQIS) Access Roles and Profile (HARP)

Code repositories are a file archive and web hosting facility providing secure storage for code and version control. 

A code system is a managed collection of concepts with each concept represented by at least one internally unique code and a human readable description, e.g., SNOMED CT.

A coding system is the symbolic arrangement of data or instructions in a computer program or the set of such instructions.

A comment period is the period of time the public has to respond to a request for public comment, such as a proposed rule, Request for Information, a new measure posted on the Measures Management System Hub, or some other document. The Administrative Procedure Act requires federal agencies to give the public an opportunity to participate in rulemaking. Executive Orders 12866 and 13563 provide guidance noting a comment period generally should be no less than 60 days, but the length of the comment period varies if not part of the rulemaking process.

Communication activities are the study and use of communication strategies to inform and influence individual and community decisions affecting health. Health communications link the fields of communication and health and are increasingly recognized as necessary elements to improve personal and public health. Examples of communication activities include traditional (e.g., print through manuscripts, professional presentations, issue briefs, white papers, television, radio media) and non-traditional (e.g., social marketing techniques and social media platforms) methods of communication to present and disseminate information, such as clinical guidelines to encourage their use and adherence. For more information see Centers for Disease Control and Prevention's Gateway to Health Communication.

A composite measure is a measure containing two or more individual measures, resulting in a single measure with a single score.

Computable care guidelines are the expression of and sharing of health care guidelines in a grammar understood by a software application. Integrating the Health Enterprise. (n.d.). Computable care guidelines. Retrieved March 12, 2024, from https://wiki.ihe.net/index.php/Computable_Care_Guidelines

Computer code is the symbolic arrangement of data or instructions in a computer program.

Concept maps are visual representations of information that can take the form of charts, graphic organizers, tables, flowcharts, Venn Diagrams, timelines, or T-charts. The Learning Center - University of North Carolina at Chapel Hill. (n.d.). Concept maps. Retrieved March 20, 2024, from https://learningcenter.unc.edu/tips-and-tools/using-concept-maps/

A continuous variable is a measure score in which each individual value for the measure can fall anywhere along a continuous scale and aggregated using a variety of methods such as the calculation of a mean or median (for example, mean number of minutes between presentation of chest pain to the time of administration of thrombolytics).

The Clinical Quality Framework (CQF) is a collaborative community of participants from the public and private sectors focused on providing tools, services, and guidance to harmonize standards for clinical decision support and electronic clinical quality measures use in clinical quality improvement.

The Clinical Quality Framework (CQF) Ruler comprises a collection of plugins tailored for the HAPI Fast Healthcare Interoperability Resources® (FHIR®) JPA server providing an implementation of the FHIR Clinical Reasoning Module for processing quality measures and serves as a knowledge artifact repository and clinical decision support service.

Clinical Quality Language (CQL) is a Health Level Seven International® mixed normative/standard for trial use and it is part of the effort to harmonize standards between electronic clinical quality measures and clinical decision support. CQL provides the ability to express logic that is human readable yet structured enough for processing a query electronically.

The Clinical Quality Language (CQL) Evaluation Engine is an open source Java-based evaluation engine capable of evaluating the result of any CQL expression. Note: this is a component in the CQL translator repository.

Clinical Quality Language (CQL) Execution Framework is a set of JavaScript libraries that can execute CQL artifacts expressed as JavaScript Object Notation Expression Logical Model.

The Clinical Quality Language (CQL) Formatting and Usage Wiki is a collaborative workspace for the development of CQL formatting conventions and usage patterns for the representation of logic within quality measures. All users have editing rights, able to submit edits, add comments and concerns. Items on the Wiki are a work in progress and subject to change.

A Clinical Quality Language (CQL) library is a container for artifact logic/expressions. A CQL library contains declarations which specify the items found in the library. Every electronic clinical quality measure has at least one CQL library. For more information, see the CQL Style Guide and the Health Level Seven International® CQL Author’s Guide.

Clinical Quality Language (CQL) Runner is an online platform for ad hoc testing of CQL.

The CQL-to-ELM Translator is an open-source reference implementation that translates the high-level Clinical Quality Language (CQL) syntax into the Expression Logical Model (ELM) representation. The reference implementation is used in support of Clinical Quality Framework implementations as a tool to enable CQL output to be uniformly and automatically translated into ELM XML or JavaScript Object Notation (JSON) documents for sharing and distribution to support implementation, integration, translation, and execution of CQL-based artifacts.

For more information or to get involved, visit the Clinical Quality Framework initiative page on the Health Level Seven International® website.

A CQMCommonQDM Library (formerly known as Global Common Library) is a shared Clinical Quality Language (CQL) library which contains CQL expressions for all eCQM developers to use when specifying electronic clinical quality measures. For more information, see the CQL Style Guide located the on the Get Started with eCQM Development tab.

A Critical Access Hospital (CAH) is a hospital in a federal program established in 1997 as part of the Balanced Budget Act designed to promote rural health planning, network development, and improve access to health services for rural residents. CAHs represent a separate provider type with their own Medicare Conditions of Participation (CoP) and a separate payment method. The Code of Federal Regulations lists the CoPs for CAHs at 42 CFR 485 subpart F.

Crucible is a suite of open-source testing tools for Health Level Seven International® Fast Healthcare Interoperability Resources® (FHIR®). It helps the FHIR development community ensure correct FHIR implementations. Crucible can test for conformance to the FHIR standards, score patient records for completeness, generate synthetic patient data, and test SMART on FHIR applications.

Cypress, with the fully integrated Cypress Validation Utility + Calculation Check (CVU+), is an open source testing tool used by vendors to certify their electronic health records (EHRs) and health information technology (IT) modules for calculating electronic clinical quality measures (eCQMs). Cypress is an official testing tool for the ONC Health IT EHR Certification Program. Testing involves Cypress generating synthetic patient records for the subset of published eCQMs selected for certification and testing the ability of the health IT modules to accurately record, import, calculate, filter, and report eCQMs.

The Cypress Tech Talks are an open forum for discussing technical implementation and tool issues with the Cypress support team.

Data capture, or electronic data capture, is the process of extracting information from a paper or electronic document and converting it into data readable by a computer. Hyland. (n.d.). What is data capture? Retrieved March 20, 2024, from https://www.hyland.com/en/resources/terminology/data-capture/what-is-data-capture

A data element in quality measurement or improvement refers to a specific piece of information that is collected and stored in a healthcare system. Each data element represents a single unit of data and can include various types of data such as patient demographics, clinical measurements, lab results, and treatment details.

Data element validity is the extent to which the information represented by the data element or code used in the measure reflects the actual concept or event intended. For example, use of a medication code as a proxy for a diagnosis code and data element response categories that include all values necessary to provide an accurate response.

See also measure validity.

A data model is an abstract model organizing elements of data and standardizing how they relate to one another. For instance, a data model linking guideline information with clinical data for the patient. Taylor, D. (2023). Data modelling: Conceptual, logical, physical model types. Retrieved March 20, 2024, from https://www.guru99.com/data-modelling-conceptual-logical.html

The De Novo Measure Scan (DNMS) is an advanced feature of the Environmental Scan Support Tool (ESST) on the controlled access CMS Measures Inventory Tool (CMIT) site. You must have a CMIT login to access. The DNMS helps measure developers conduct early and frequent environmental scans while developing new measures. The DNMS uses a Clinical Quality Measure ontology to describe a measure and represent quality measure concepts. Components of the ontology include target population, health status or utilization, change concept, expected output of the change concept, and care setting. The DNMS uses structured search terms to represent the measure concepts to build a new measure and uses artificial intelligence to identify and prioritize the most relevant PubMed, PubMed Central, and CINAHL related literature.

Decision Model and Notation (DNM) is a standard published by the Object Management Group. It is a standard approach for describing and modeling repeatable decisions within organizations to ensure decision models are interchangeable across organizations. Oliveira, W. (2018, August 21) What is decision model and notation (DMN)? Retrieved March 20, 2024, from https://www.heflo.com/blog/process-modeling/decision-model-and-notation-dmn/

The denominator is the lower part of a fraction used to calculate a rate, proportion, or ratio. It can be the same as the initial population or a subset of the initial population to further constrain the population for the purpose of the measure. Continuous variable measures do not have a denominator, but instead define a measure population.

A denominator exception is any condition that should remove a patient, procedure, or unit of measurement from the denominator of the performance rate only if the numerator criteria are not met. A denominator exception allows for adjustment of the calculated score for those measured entities with higher risk populations. A denominator exception also provides for the exercise of clinical judgment and the measure developer should specifically define where to capture the information in a structured manner that fits the clinical workflow. The measured entity removes denominator exception cases from the denominator. However, the measured entity may still report the number of patients with valid exceptions. Allowable reasons fall into three general categories - medical reasons, patient reasons, or system reasons. Only proportion measures use a denominator exception.

A denominator exclusion is a case the measured entity should remove from the measure population and denominator before determining if numerator criteria are met. Proportion and ratio measures use denominator exclusions to help narrow the denominator. For example, a measure developer would list patients with bilateral lower extremity amputations as a denominator exclusion for a measure requiring foot exams. Continuous variable measures may use denominator exclusions but may use the term measure population exclusion instead of denominator exclusion.

The Health Level Seven International® (HL7®) Data Exchange for Quality Measures (DEQM) Implementation Guide (IG), provides a framework defining conformance profiles and guidance to enable the exchange of quality improvement information including quality measure reporting (e.g., for transferring quality information from a health care provider to a payor) using Fast Healthcare Interoperability Resources® (FHIR®)-based resources.

The DEQM IG supports data exchange and reporting scenarios including individual reporting, summary reporting, and gaps in care reporting. The DEQM Individual MeasureReport profile supports individual reporting and the DEQM Summary MeasureReport profile supports summary reporting. These profiles support the Quality Reporting Document Architecture (QRDA) Category I and QRDA Category III reporting use cases in the Quality Data Model-Clinical Quality Language-QRDA paradigm.

The HL7 workgroup, Clinical Quality Information (CQI), manages the changes to this FHIR IG through the standard balloting process. The update cycle follows closely with the Quality Measure IG and Quality Improvement Core (QI-Core) Implementation Guide (IG). 

Derivative products, with respect to clinical practice guidelines, are products with content derived from the content of the practice guideline, e.g., clinical decision support, patient/family guides, pocket cards, phone apps for clinicians, continuing education programs.

A digital platform is an established device or web-based platform for presenting cloud technology and content, such as Facebook, X (formerly Twitter), Blogs, Websites, and sometimes short message service. This is in contrast to an analog platform, e.g., billboards, direct mail, telemarketing, events, word-of-mouth.

CMS defines digital quality measures (dQMs) as quality measures that use standardized, digital data from one or more sources of health information that are captured and exchanged via interoperable systems; apply quality measure specifications that are standards-based and use code packages; and are computable in an integrated environment without additional effort.

A direct reference code (DRC) is a specific code referenced directly in the electronic clinical quality measure logic to describe a data element or one of its attributes. DRC metadata include the description of the code, the code system including the code, and the version of that code system.

The Electronic Clinical Quality Improvement (eCQI) Resource Center User Group is a volunteer forum of eCQI Resource Center users that use the site to obtain resources necessary for electronic clinical quality measure (eCQM) development, implementation, reporting, and education on key eCQM and eCQI concepts and standards. The eCQI Resource Center User Group meets quarterly on the 3rd Tuesday at 3:00pm ET.

This document describes the versions of the standards and code systems used in conjunction with the updated eCQMs for potential use in CMS programs for a given reporting/performance period.

The Measure Collaboration Workspace Electronic Clinical Quality Measures (eCQM) Concepts Module gives users the ability to suggest new eCQM concepts and comment on other's suggested eCQM concepts.

CMS encourages all stakeholders, including patients and point of care clinicians, to suggest eCQM concepts and offer comments on others' suggested eCQM concepts. 

The eCQM Data Element Repository (DERep) provides clarification, definitions, and clinical focus for the data elements associated with electronic clinical quality measures (eCQMs) used in CMS quality reporting and incentive programs. Users can filter information by data element, eCQM, Quality Data Model (QDM) attribute, QDM category, QDM datatype, or QDM entities.

The electronic clinical quality measure (eCQM) Flows are flowcharts designed to assist in the interpretation of the eCQM logic and provide the calculation methodology to determine measure scores. These flows provide an overview of each of the population criteria components and associated data elements that lead to the inclusion, exclusion, or exception into the eCQM’s denominator and numerator, if applicable.

The eCQM flows are available for eligible clinician eCQMs, hospital-inpatient eCQMs, and hospital-outpatient eCQMs.

The Electronic Clinical Quality Measure (eCQM) Implementation Checklist assumes a health care practice/organization has determined which quality measures to report. The checklist provides the necessary technical steps health information technology developers, implementers, and health care organizations must take to update their systems and processes with the eCQM Annual Update for the upcoming reporting and performance periods. The most recent eCQM Annual Update should be applied to electronic health record systems for use in electronic quality reporting.

The Electronic Clinical Quality Measure (eCQM) Known Issues tracker provides implementation information for eCQMs with known technical issues for which a solution is under development but not yet available in a published eCQM specification. This includes discrepancies between eCQM narrative and logic, value sets, and/or technical, standard, or logic-related issues.

The Electronic Clinical Quality Measure (eCQM) Logic and Implementation Guidance is a resource document which provides guidance for understanding, using, and/or implementing eCQMs. This document includes useful context and details regarding topics such as Clinical Quality Language representation in the technical specifications, applying data elements and value sets/coding, and the Assistant Secretary for Technology Policy/Office of National Coordinator for Health Information Technology (ASTP/ONC) Jira to provide feedback on the eCQMs.

The Electronic Clinical Quality Measure (eCQM) Testing Opportunities Module within the eCQI Resource Center’s Measure Collaboration Workspace enables eCQM developers to announce testing opportunities. This module allows developers to evaluate various aspects of eCQMs, including feasibility, reliability, validity, and usability. Interested parties wanting to participate in testing eCQMs should contact the eCQM developer directly.

An efficiency measure is the cost of care (inputs to the health system in the form of expenditures and other resources) associated with a specified level of health outcome.

Electronic case reporting is the automated generation and electronic submission of reportable diseases and conditions from an electronic health record to public health agencies. Each state has public health reporting requirements and relies on health care providers to report on certain conditions.

Electronic clinical quality improvement is the use of health information technology, and the functionality and data in an electronic health record and/or other health information technology, along with clinical best practices to support, leverage, and advance quality improvement initiatives.

An electronic clinical quality improvement (eCQI) implementer does many things to prepare processes and systems such as

• Putting measure data components into systems and workflow
• Using measures when conducting health care activities
• Providing information from measures to inform quality improvement (e.g., health information technology implementer, quality analyst, quality reporting validator)

An electronic clinical quality measure (eCQM) is a measure specified in a standard electronic format that uses data electronically extracted from electronic health records (EHR) and/or health information technology (IT) systems to measure the quality of health care provided.

An electronic health record (EHR) is also known as the electronic patient record, electronic medical record, or computerized patient record. An EHR is a “longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting. Included in this information are patient demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data, diagnoses and treatment, medications, allergies, immunizations as well as radiology images and laboratory results.” International Social Security Association. (n.d.). Information and communication technology- Guideline 91. Electronic health record system. Retrieved March 20, 2024, from https://ww1.issa.int/guidelines/ict/180156

An eligible clinician refers to a clinician who is eligible to participate in the Quality Payment Program through the Merit-based Incentive Payment System and similar participants of other CMS programs using electronic clinical quality measures for quality reporting such as Alternative Payment Model participants.

An end user (sometimes end-user) is a person who ultimately uses or intends to ultimately use a guideline, measure, or its derivative products.

An environmental scan is the process of systematically reviewing and interpreting data to identify issues and opportunities influencing prioritization of current or future plans.

The Environmental Scan Support Tool (ESST) is a tool within the CMS Measures Inventory Tool (CMIT) environment, intended to automate environmental scans required in the information gathering process to develop and maintain quality measures. The ESST uses an automated natural language processing approach that rapidly scans literature in PubMed, PubMed Central, and CINAHL to

• Identify relevant documents (abstracts and full-text articles)
• Identify and extract the specific knowledge within each relevant document that applies to the measure’s potential opportunity for improvement

The Expression Logical Model (ELM) is a machine-readable representation of an electronic clinical quality measure’s logic and provides the information needed to automatically retrieve data from an electronic health record. The ELM file can be in eXtensible Markup Language (XML) (.xml) or JavaScript Object Notation (JSON) (.json).

eXtensible Markup Language (XML) is a markup language defining a set of rules for encoding documents in both a human-readable and machine-readable format.

An extensional value set is a set of concept codes and descriptors, in the form of an enumerated list, selected to serve a specific purpose.

A Fast Healthcare Interoperability Resources® (FHIR®) profile is a set of requirements and constraints placed on a resource. It can describe general features the system supports for that resource or information handled or produced according to a specific use case. Often, they include rules about which application programming interface features, terminologies, or resource elements the FHIR profile uses or does not use.

The feasibility criterion assesses the extent to which the specifications, including measure logic, require readily available data or could be captured without undue burden and can be implemented for performance measurement. 

The Federal Register is the official daily publication for rules, proposed rules, and notices of federal agencies and organizations, as well as executive orders and other presidential documents. The Federal Register is published by the Office of the Federal Register, National Archives and Records Administration.

Fast Healthcare Interoperability Resources® (FHIR®) is an open-source Health Level Seven International® standard for exchanging health care information electronically. It is the next generation exchange framework being adopted by the health care community to advance interoperability.

The Fast Healthcare Interoperability Resources® (FHIR®) Electronic Clinical Quality Measure (eCQM) Examples is a webpage in the Quality Measure Implementation Guide containing example eCQMs using FHIR.

The Fast Healthcare Interoperability Resources® (FHIR®) Implementation Guide Registry is a list of many of the implementation guides defined by the FHIR community.  

The Fast Healthcare Interoperability Resources® (FHIR®) Terminology Service for Value Set Authority Center (VSAC) Resources is a representational state transfer application programming interface service for accessing the VSAC value sets and supported code systems. It assumes working knowledge of FHIR resources. Access requires a free Unified Medical Language System (UMLS) license.

Future state is a model integrating the downstream work of informatics, implementation, communication, and evaluation into the guideline or measure development process.

Git is a free and open source distributed version control system for tracking changes in any set of files among programmers cooperating on source code during software development. It is designed to handle everything from small to very large projects with speed and efficiency.

GitHub, Inc. is a subsidiary of Microsoft, which provides hosting for software development and version control using Git.

GRADE Evidence to Decision framework is a systematic and transparent approach to making well-informed health care choices. By using the GRADE EtD framework, healthcare organizations can make more consistent and evidence-based decisions about quality measures, which helps improve the overall quality of care provided to patients.

Grading of Recommendations Assessment, Development, and Evaluation (GRADE) is a transparent framework for developing and presenting summaries of evidence and provides a systematic approach for making clinical practice recommendations. It is the most widely adopted tool for grading the quality of evidence and for making recommendations.

The Guide for Reading Electronic Clinical Quality Measures (eCQMs) is a resource to help measured entities, quality analysts, implementers, and health information technology vendors understand eCQMs. The Guide provides background on eCQM packages, which are the building blocks of eCQMs, and an overview for a user to understand the human-readable format of eCQMs. The Guide also provides information related to value sets, Quality Data Model data criteria, and more.

Guideline evaluation activities are a systematic collection of information about a guideline. These activities involve collecting and analyzing information about a guideline's activities, characteristics, and outcomes to make judgments about the guideline, improve its effectiveness, and/or inform decisions about future guideline development.

Guideline Implementation with Decision Support (G.U.I.D.E.S.) is a tool to assist professionals when implementing guidelines with clinical decision support.

A guideline oversight or steering committee, selected at the beginning of a project, is an advisory body of senior interested parties or experts that gain leadership support and are accountable for the proposed guideline. This committee provides guidance on issues emerging in the guideline development process, including the products emerging from all guideline working groups.

A guideline recommendation tells the intended end-user of a guideline what they can or should do in specific situations to achieve the best health outcomes possible, individually or collectively. It offers a choice among different interventions or measures having an anticipated positive impact on health and implications for the use of resources. 

Guideline working groups include experts who conduct the guideline development work. Working groups can focus on different areas of expertise such as conducting the systematic literature review, developing the computable guideline, designing the implementation process, communicating/disseminating the guideline, or evaluating the impact of the guideline.

Harmonization is the standardization of specifications for related measures with the same measure focus (for example, influenza immunization of patients in hospitals or nursing homes); related measures for the same target population (for example, eye exam and HbA1c for patients with diabetes); or definitions applicable to many measures (for example, age designation for children) so they are uniform or compatible, unless differences are justified (in other words, dictated by the evidence). The dimensions of harmonization can include numerator, denominator, exclusions, calculation, and data source and collection instructions. The extent of harmonization depends on the relationship of the measures, the evidence for the specific measure focus, and differences in data sources. Measure developers should harmonize value sets used in measures when the intended meaning is the same. Harmonization can also mean adoption of the same standard(s) for different purposes such as use of standards for electronic clinical quality measures (eCQMs) and clinical decision support (CDS), for example Clinical Quality Language for logic expression in eCQMs and CDS.

A health care consumer is an individual who uses the services of a health care provider including patients receiving medical care or treatment. IGI Global Services. (n.d). What is health consumer. Retrieved March 20, 2024, from https://www.igi-global.com/dictionary/empirical-study-patient-willingness-use/33258

A health care organization is a purposefully designed, structured social system developed for the delivery of health care services by specialized workforces to defined communities, populations, or markets.

A health care payor is any payer of health care services other than the insured person, e.g., insurance company, Health Maintenance Organization, Preferred Provider Organization, or the federal government.

Health information technology (health IT) involves the processing, storage, and exchange of health information in an electronic environment. In the Health Information Technology for Economic and Clinical Health (HITECH) Act, the term health information technology includes hardware, software, integrated technologies or related licenses, intellectual property, upgrades, or packaged solutions provided as services designed for or support the use by health care entities or patients for the electronic creation, maintenance, access, or exchange of health information.

The Office of the National Coordinator for the Health Information Technology (ONC) Certification Program is a voluntary certification program established by ONC to provide for the certification of health information technology. The program ensures that Certified Health Technology meets the technological capability, functionality, and security requirements adopted by the U.S. Department of Health and Human Services. The ONC develops the functional and conformance testing requirements for the testing and certification criteria related to electronic clinical quality measures (eCQMs).

A health information technology (IT) developer/vendor is an entity that designs, develops, and/or markets health IT software application(s) for use in hospitals, ambulatory care settings, and/or specialty care delivery settings, e.g., electronic health record (EHR) developer, EHR vendor, quality reporting software developer, quality reporting software vendor.

Health Level Seven International is a standards-developing organization providing a framework and international standards for the exchange, integration, sharing, and retrieval of electronic health information (including clinical and administrative data) to support clinical practice and the management, delivery, and evaluation of health services. These standards for transfer of data between health care software applications focus on the application layer, which is "layer 7" in the Open Systems Interconnection model (OSI model), a conceptual model characterizing and standardizing the communication functions of a telecommunication or computing system without regard to its underlying internal structure and technology.

Healthcare Service Location (HSLOC) is a classification of locations and settings where health care services are provided. HSLOC can be used to identify locations within a facility, such as the emergency department and medical intensive care unit.

This Health Level Seven International® (HL7®) Clinical Quality Language (CQL)-Health Quality Measure Format (HQMF) Implementation Guide (IG) defines an approach to using CQL with the Quality Data Model (QDM) and HQMF to define electronic clinical quality measures (eCQMs). The IG describes the use of CQL to express the logic and the US Domain conceptual data model, QDM, to express the individual data types and their attributes for an eCQM detailed using the HQMF standard.

The Health Level Seven International® (HL7®) Quality Reporting Document Architecture Category I Implementation Guide (HL7 QRDA I IG) describes constraints on HL7's Clinical Document Architecture Release 2 (CDA R2) to create a QRDA I document. QRDA Category I is for reporting quality data at the individual patient level.

The Health Level Seven International® (HL7®) Quality Reporting Document Architecture Category III Implementation Guide (HL7 QRDA III IG) describes constraints on HL7's Clinical Document Architecture Release 2 (CDA R2) to create a QRDA III document. QRDA Category III is for reporting aggregated quality measure data and supports reporting for improvement activities and Promoting Interoperability measures.

The Quality Measure Implementation Guide (IG), or QM IG, is intended to provide guidance on the representation of quality measures including electronic clinical quality measures (eCQMs) using Fast Healthcare Interoperability Resources® (FHIR®) Clinical Reasoning Module. It provides requirements for eCQM structure, metadata, definitions, and logic for use when authoring, packaging and evaluating quality measures.

An inpatient hospital is an acute care facility, e.g., Subsection (d) hospitals in the 50 states or District of Columbia paid under the Inpatient Prospective Payment System, and critical access hospitals, meeting eligibility requirements for Promoting Interoperability Program payment adjustments by adopting, implementing, or updating certified EHR technology.

An outpatient hospital refers to a medical or surgical care facility classified as a hospital that is usually paid under the Outpatient Prospective Payment System (OPPS), where patients receive services for conditions that do not require an overnight inpatient stay. Outpatient care refers to numerous types of health services, such as emergency department (ED), observation, surgical, diagnostic, and therapeutic services.

The Hospital Quality Reporting (HQR) Secure Portal is the only CMS-approved website for secure communications and health care quality data exchange between quality improvement organizations, hospitals, physician offices, nursing homes, end-stage renal disease networks and facilities, and data vendors. It provides secure data exchange to and within various CMS quality reporting programs, including Hospital Inpatient Quality Reporting, Hospital Outpatient Quality Reporting, Inpatient Psychiatric Facility Quality Reporting, Prospective Payment System-Exempt Cancer Hospital Quality Reporting, Ambulatory Surgical Center Quality Reporting, and the Medicare Promoting Interoperability Program.  

For hospital quality reporting, users must create a Health Care Quality Information Systems (HCQIS) Access Roles and Profile (HARP) account to log into HQR. Submitters can navigate within the HQR system and perform tasks based on the program data that are submitted, such as data submissions, data results, program reporting, and administration. For information on registering for a HARP account and additional resources, visit the Registering for HARP page on QualityNet. Please contact the Centers for Clinical Standards and Quality (CCSQ) Service Center at (866) 299-8912 or QNetSupport@cms.hhs.gov for assistance.

HL7 defines the Health Quality Measures Format (HQMF) as a standard for representing a health quality measure as an electronic document. Quality measures encoded in the HQMF format are referred to as electronic clinical quality measures, or eCQMs.

A hybrid measure is a quality measure using more than one source of data for measure calculation. Current hybrid measures use claims data and electronic clinical data from electronic health records to calculate measure results.

Hypertext Markup Language (HTML) is the standard markup language for creating web pages and web applications.

The importance criterion assesses the extent to which the specific measure focus is evidence-based and important to making significant gains in health care quality (e.g., safety, timeliness, effectiveness, efficiency, equity, patient-centeredness) and improving health outcomes for a specific high-impact aspect of health care where there is variation in or overall poor performance. 

Industry standard modeling techniques use a standard visual approach to facilitate communication related to business models and software design. For example, Unified Modeling Language (UML) uses integrated diagrams to specify and document the structure and dynamic behavior of information system and software artifacts.

Inferno is an open-source tool designed for creating, executing, and sharing automated conformance tests for the Fast Healthcare Interoperability Resources® (FHIR®) standard. Hosted on HealthIT.gov, Inferno offers a public service for running select FHIR conformance tests, including those relevant to the ONC Health IT Certification Program. It features the ONC Certification (g)(10) Standardized API Test Kit and tests for various FHIR Implementation Guides. Inferno provides flexible tools for testing systems for FHIR conformance, allowing users to run tests locally or integrate them into external tools and continuous integration pipelines.

Informatics activities include tasks related to knowledge engineering and technical development of clinical practice guidelines artifacts (L2 to L4)(See Levels of knowledge representation for CPG). These may include, but not limited to developing use case diagrams, flow diagrams, and decision trees; identifying relevant data elements, Fast Healthcare Interoperability Resources® profiles, terminologies, value sets, libraries, rules, and logic representations; writing software code (e.g., for clinical decision support); and developing comprehensive test cases and testing resources.

With an integrated process, steps related to guideline development, informatics activities, implementation, evaluation, and communications are conducted in parallel, insofar as possible, rather than in sequence.

An intensional value set is a list of codes based on a logical statement that often has an algorithmic basis for selection of concepts.

An interested party is an individual, group, or organization affected by the outcome of a project and, thus, has an interest in the project's success.

An intermediate outcome measure is a measure assessing the change produced by a health care intervention leading to a long-term outcome.

An inverse measure is a measure in which a lower performance rate is better. Therefore, a zero performance rate for these measures is a good score. For example, the National Healthcare Safety Network calculates most healthcare-associated infections (HAIs) as a standardized infection ratio (SIR). The SIR compares the actual number of HAIs (the numerator) with the predicted number based on the baseline U.S. experience (e.g., standard population), adjusting for several risk factors found to be most associated with differences in infection rates. The goal is to have the numerator equal to or very close to zero, thereby having an SIR equal to or very close to zero.

The Interoperability Standards Advisory (ISA) process represents the model by which the Assistant Secretary for Technology Policy (ASTP) coordinates interoperability standards and implementation specifications for industry use to fulfill specific clinical health information technology interoperability needs. ASTP updates the ISA throughout the year to include improvements based on recommendations received from public comments and subject matter experts. These updates are published in an annual Reference Edition. To learn more about the ISA, please visit https://www.healthit.gov/isp/isp/about-isa.

JavaScript Object Notation (JSON) is a lightweight data-interchange format. It is easy for humans to read and write, easy for machines to parse and generate, and is based on a subset of the JavaScript Programming Language Standard ECMA-262 3rd Edition - December 1999. JSON is a completely language-independent text format but uses conventions familiar to programmers of the C-family of languages including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language. JSON.org. (n.d.). Introducing JSON. Retrieved March 20, 2024, from https://www.json.org/json-en.html.

Knowledge assets (e.g., declarative or models) refer to specific knowledge-related items. These can include declarative (factual) assets or procedural assets describing how facts will be used. Types of knowledge assets can include data, expertise, lessons learned, policies and procedures, and other knowledge-related documents. Freeze, R.D. & Kulkarni, U. (2007). Knowledge management capability: Defining knowledge assets. Journal of Knowledge Management,11(6), pp. 94-109. https://doi.org/10.1108/13673270710832190

A knowledge management platform automatically structures and organizes content to be quickly findable, navigable, and searchable in a variety of places. 

There are four levels of knowledge representation for clinical practice guidelines (CPGs).

• L1 Narrative: The knowledge is human-readable and in an unstructured format. This knowledge can be used as an input for crafting policies or making decisions. The structure does not allow machines to utilize knowledge.
• L2 Semi-structured: This level has some structure, paving the way for clinical domain experts to interpret. HUMAN READABLE
• L3 Structured: This adds a machine-interpretable structure to the knowledge representation. Knowledge can be shared across settings and systems. There may be subjectivity in knowledge interpretation. COMPUTER READABLE
• L4 Executable: This format enables not only machine-readable knowledge representation, but also machine-executable knowledge representation based on the patient’s clinical context with a specific type of clinical decision support tool. CLINICAL DECISION SUPPORT

A living guideline uses the results of a living systematic review and determines whether a guideline needs new guideline recommendations or needs modifications to existing guideline recommendations as new evidence emerges. Akl, E. A., Meerpohl, J. J., Elliott, J., Kahale, L. A., Schünemann, H. J., & the members of Living Systematic Review Network. (2017). Living systematic reviews: 4. Living guideline recommendations. Journal of Clinical Epidemiology, 91, 47-53. https://doi.org/10.1016/j.jclinepi.2017.08.009

A living systematic review uses the same processes as other systematic reviews but is continually updated, based on frequent searches of the literature, with the incorporation of relevant new evidence as it becomes available. Cochrane Community. (n.d.) Living systematic reviews. Retrieved March 20, 2024, from https://community.cochrane.org/review-production/production-resources/living-systematic-reviews

A logic model is a graphic depiction (road map) representing the shared relationships among the resources, activities, outputs, outcomes, and impact for the evaluation of a proposed guideline. This visual framework shows the critical logical premises and presumed relationships among intermediate, surrogate, and ultimate health outcomes related to a specified clinical question. Woolf, S., Schünemann, H. J., Eccles, M. P., Grimshaw, J. M., & Shekelle, P. (2012). Developing clinical practice guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations. Implementation Science, 7(1), 1-12. https://doi.org/10.1186/1748-5908-7-61

Machine learning is a branch of artificial intelligence and computer science focusing on the use of data and algorithms to imitate human learning, gradually improving in accuracy. IBM. (n.d.). What is machine learning? Retrieved March 20, 2024, from https://www.ibm.com/topics/machine-learning

The Measure Authoring Development Integrated Environment (MADiE) is a software tool redefining the electronic clinical quality measure (eCQM) development and testing process by making it a self-contained process including dynamic authoring and testing within a single application. MADiE allows eCQM developers to author eCQMs using Quality Improvement (QI)-Core, the Quality Data Model (QDM), Fast Healthcare Interoperability Resources® and to test and verify eCQM behavior. MADiE helps eCQM developers execute eCQM logic against the constructed test cases and evaluates if the eCQM logic aligns with the intent of the eCQM.

The Measure Authoring Development Integrated Environment (MADiE) User Group is an open forum to discuss the status of the tool and gather community and user feedback on planned enhancements and releases. The User Group is scheduled to meet on the third Thursday of every month. Register for the MADiE User Group via the meeting appointment.

The Measure Collaboration (MC) Workspace is a web-based tool bringing together a set of interconnected resources, tools, and processes to promote transparency and better interaction across interested parties that develop, implement, and report electronic clinical quality measures.

CMS’s Meaningful Measures Initiative identifies high priority areas for quality measurement and improvement, with the goal of improving health outcomes for patients, their families, and measured entities (e.g., clinicians, hospitals). Its purpose is to deliver value by empowering patients to make informed care decisions while also reducing burden on measured entities.

A measure developer is an individual or organization responsible for the development, implementation, and maintenance of a measure. Measure developers may create, edit, and submit measures for consideration by CMS to include in programs. CMS encourages measure developers to use the Blueprint content on the Measures Management System Hub as a guide in creating their measures and to collaborate with other measure developers to share best practices/new learnings freely, e.g., CMS measure development contractors, hospital systems, medical associations, or federal health agencies.

The measure observation is the computation reporting entities should perform on the members of the measure population after removing the measure population exclusions. Only continuous variable measures use measure observation.

The measure score is the numeric result computed by applying the measure specifications and scoring algorithm. The computed measure score represents an aggregation of all appropriate patient-level (for example, proportion of patients who died, average lab value attained) or episode-level data (for example readmission measures) for the measured entity (hospital, health plan, home health agency, clinician, etc.). The measure specifications designate the measured entity and to whom the measure score applies.

A measure steward is an individual or organization that owns a measure and is responsible for maintaining the measure. Measure stewards may also be measure developers. Measure stewards are also the ongoing point of contact for people interested in a given measure e.g., medical specialty society or federal health agency.

Measure testing is empirical analysis to assess the evaluation criteria (e.g., importance, feasibility, scientific acceptability - reliability and validity, usability and use) of a measure as specified. It includes analysis of issues posing threats to the validity of conclusions about quality of care such as exclusions, risk adjustment/stratification for outcome and resource use measures, methods to identify differences in performance, and comparability of data sources/methods.

The Measures Under Consideration (MUC) List is a list of quality and efficiency measures the Department of Health & Human Services is considering adopting, through the federal rulemaking process, for use in the Medicare program. The MUC list is made publicly available by December first each year for categories of measures described in section 1890(b) (7) (B) (i) (I) of the Social Security Act (SSA) as amended by Section 3014 of the Patient Protection and Affordable Care Act.    https://mmshub.cms.gov/measure-lifecycle/measure-implementation/pre-rulemaking/overview

Measure validity is when the measure accurately represents the evaluated concept and achieves the intended purpose (to measure quality). For example, the measure

• Clearly identifies the evaluated concept (face validity)
• Includes all necessary data elements, codes, and tables to detect a positive occurrence when one exists (construct validity)
• Includes all necessary data sources to detect a positive occurrence when one exists (construct validity)

Measured entities are the front-line clinicians, including health information technology professionals, and their organizations, who collect quality measurement data. Measured entities are the implementers of quality measures. The effect of quality measure data collection on clinician workflow can be negative. There may be effects on their payments, positive and negative, with respect to reporting and actual performance on quality measures. Because of these potential effects, measured entities should be involved in all aspects of the Measure Lifecycle.

The Merit-based Incentive Payment System (MIPS) assigns the MIPS Quality Identification to a quality measure in use in MIPS. CMS uses the MIPS Quality identification (ID) in MIPS documentation including Physician Payment System proposed and final rules.

The Measures Management System (MMS) is a standardized system for developing and maintaining the quality measures used in CMS’s various quality initiatives and programs. The primary goal of the MMS is to provide guidance to measure developers to help them produce high-caliber health care quality measures.

A narrative guideline is a text-based representation of the practice guideline. A narrative guideline typically includes guideline recommendations and additional explanatory text.

A needs assessment is aimed at delineating differences between the current state and the ideal state, followed by determining why these gaps exist and identifying solutions to address them.

The null performance rate is when all of the denominator eligible instances are attributed to all denominator exceptions. Therefore, the performance rate for satisfactory reporting would be 0/0 (null).

The numerator is the upper portion of a fraction used to calculate a rate, proportion, or ratio. Also called the measure focus, it is the target process, condition, event, or outcome. Numerator criteria are the processes or outcomes expected for each patient, procedure, or other unit of measurement defined in the denominator. A numerator statement describes the action that satisfies the conditions of the quality measure.

A numerator exclusion defines an instance measured entities should not include in the numerator data. Use numerator exclusions only in ratio and proportion measures.

An ontology is a machine-readable, formal representation of knowledge within a domain but also understandable to humans. It typically includes unique hierarchically arranged concepts that have specific attributes and are semantically related to other concepts. Dissanayake, P. I., Colicchio, T. K., & Cimino, J. J. (2020). Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis. Journal of the American Medical Informatics Association, 27(1), 159-174. https://doi.org/10.1093/jamia/ocz169

An outcome measure is a measure focusing on the health status of a patient (or change in health status) resulting from health care - desirable or adverse.

A parameter estimate provides a statistical approximation of a specific measure describing a population.

A patient-reported outcome (PRO) is a status report of a patient’s health condition or health behavior that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else. This definition reflects the key domains of

• Health-related quality of life (including functional status)
• Symptoms and symptom burden (e.g., pain, fatigue)
• Health behaviors (e.g., smoking, diet, exercise)

(Adapted from the Food and Drug Administration. (2009). Guidance for industry PRO measures: Use in medical product development to support labeling claims. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-reported-outcome-measures-use-medical-product-development-support-labeling-claims) 

A patient-reported outcome measure (PROM) is an instrument, scale, or single-item measure used for assessing a patient-reported outcome concept as perceived by the patient, obtained by directly asking the patient to self-report.

A patient-reported outcome-based performance measure (PRO-PM) is a performance measure based on patient-reported outcome measure (PROM) data aggregated for an accountable health care entity. Measured entities collect the data directly from the patient using the PROM tool, which can be an instrument, scale, or single-item measure.

Pioneers in Quality™ is a collaborative Joint Commission program to improve health care quality and patient safety through education (e.g., webinars for continuing education units), innovation, and recognition of pioneering practices for Joint Commission interested parties. Started in 2016 to assist hospitals on their electronic clinical quality measure (eCQM) journey to improve eCQM data use for quality improvement, Joint Commission has expanded the program over time to include educational topics related to quality measurement, data submission and use for quality improvement, as well as implementation of new and revised accreditation and certification standards.

A point estimate provides a single value to use from a sample to estimate the population parameter.

Population, Intervention, Comparator group, Outcomes, Time interval, Settings (PICOTS) is a framework for delineating clinical questions facilitating identification of relevant research evidence. For example, among adults with non-cancer-related pain (Population) who receive an opioid pain medication (Intervention) versus a non-opioid pain medication or placebo (Comparator), what are the effects on ratings of pain, functioning, and quality of life (Outcome) within 3 months of treatment (Time) in outpatient settings of care (Setting)? Chang, S. M. and Matchar, D. B. (Eds.). (2012, June). Methods guide for medical test reviews. AHRQ Publication No. 12-EHC017. Retrieved March 20. 2024, from https://www.ncbi.nlm.nih.gov/books/NBK98241/pdf/Bookshelf_NBK98241.pdf

Postman is a collaboration platform designed for developing and testing application programming interfaces (APIs). It serves as a hypertext transfer protocol (HTTP) client that allows users to test HTTP requests and receive various types of responses through an intuitive graphical user interface. Postman simplifies interactions with Fast Healthcare Interoperability Resources® (FHIR®) servers, enabling users to post requests and updates efficiently. It is particularly useful for testing FHIR-based electronic clinical quality measures during Health Level Seven International® Connectathons.

A process measure is a measure focusing on steps that should be followed to provide good care. There should be a scientific basis for believing the process, when executed well, will increase the probability of achieving a desired outcome.

Program Candidate measures are measures not yet adopted for use in a CMS quality reporting program. The measure specifications use versions of standards and tools designated for a specific reporting/performance period. Program Candidate measures are not eligible for CMS quality reporting until CMS proposes and finalizes through notice-and-comment rulemaking for each applicable program.

A proportion is a score derived by dividing the number of cases meeting a criterion for quality (the numerator) by the number of eligible cases within a given time frame (the denominator) where the numerator cases are a subset of the denominator cases (for example, percentage of eligible women with a mammogram performed in the last year).

Public comments ensure measures are developed and maintained using a transparent process with balanced input from relevant interested parties. Public comment periods provide an opportunity for the widest array of interested parties to provide input on the measures, both under development and in use, and can provide critical suggestions not previously considered by the measure developer or its Technical Expert Panel. Electronic clinical quality measure (eCQM) public comments can be made in response to a new eCQM's call for comments, eCQMs in the pre-rulemaking and rulemaking process, eCQMs in testing, eCQMs undergoing updates, and updates to standards and tools used with eCQMs.

Health Level Seven International® (HL7®) provides a list of Fast Healthcare Interoperability Resources® servers publicly available for testing. These are public services provided by volunteers and HL7 makes no representations concerning their safety or reliability.

The Quality Data Implementation (QDI) User Group is a group of volunteer members who use the Quality Data Model (QDM) and Fast Healthcare Interoperability Resources® (FHIR®)-related standards. The group seeks to understand data capture and use with respect to workflow in health information technology implementations such that measure requests for data are feasible and provide accurate, comprehensive, and reliable data for evaluation. The data model efforts support the electronic clinical quality improvement landscape and the evolving transition from the QDM to the FHIR model (Quality Improvement Core) for authoring electronic clinical quality measures and digital quality measures. The QDI User Group meets at 3:00 p.m. ET on the third Wednesday of every month.

The Quality Data Model (QDM) is a conceptual information model defining principal concepts (classes) of information, their related attributes, and their relationships. The Model provides a standardized way to express what needs to be retrieved from electronic records to enable quality performance measurement. The QDM is the current structure for electronically representing quality measure concepts for stakeholders involved in electronic quality measure development and reporting.

The Quality Data Model (QDM) Known Issues provides identified issues relative to working with the QDM. Many of these issues originated as tickets in the ONC Project Tracking System (Jira) QDM Issue Tracker project and all were discussed by the Quality Data Implementation Group.

The Quality Data Model (QDM)-based CQL Style Guide provides standardized expressions of electronic clinical quality measure (eCQM) concepts across eCQMs and defines a uniform “look and feel” to eCQM logic. The QDM-based CQL Style Guide focuses on a set of common best practices that have been implemented across QDM-based eCQMs in CMS quality reporting programs. The QDM-based CQL Style Guide also promotes the use of consistent language within the framework of Clinical Quality Language (CQL), including libraries, aliases, definitions, and functions, and provides guidance on other conventions, such as operator precedence. eCQM stewards or developers who are developing or specifying eCQMs for potential inclusion in CMS reporting programs should follow these best practices.

Quality Improvement Core (QI-Core) is a Health Level Seven International® (HL7®) implementation guide that defines a set of Fast Healthcare Interoperability Resources® (FHIR) profiles that support successful creation of interoperable and quality-focused applications including Electronic Clinical Quality Measures (eCQMs) and Clinical Decision Support (CDS) artifacts.

The Quality Payment Program (QPP) Developer Tools allow industry developers, who build software for participating clinicians, to integrate their software directly with CMS applications and data. Available tools include Submissions Application Programming Interface (API), Submissions Developer Preview environment, Measures, Activities, Benchmarks Repository, and Quality Reporting Document Architecture III Conversion Tool Open Source Package. These include documentation for reporting eCQMs. Additionally, there is information on submitting data through the CMS Web Interface and developer documentation on the Eligibility API.

The Quality Payment Program (QPP) Resource Library is the one-stop shop for all QPP resources. It provides educational materials supporting clinicians’ participation in QPP. All QPP webinar resources, including slides, transcripts and recordings from recent presentations, can be found on the QPP Webinar Library. Measure specifications and benchmarks are located within the Full Resource Library section of the QPP Resource Library.

The Quality Reporting Document Architecture (QRDA) is a standard document format for the exchange of electronic clinical quality measure data.

The Quality Reporting Document Architecture (QRDA) III Converter tool converts QRDA III files to the Quality Payment Program (QPP) JavaScript Object Notation format for QPP data submission. 

Quality Reporting Document Architecture (QRDA) Known Issues provide implementation information for QRDA Implementation Guides or supporting documents with known technical issues for which a solution is under development but may not be published. This includes issues such as discrepancies between conformance assertions and message text, non-critical conformance assertion errors, and typographical errors.

A qualitative assessment uses approaches such as focus groups, in-depth interviews, detailed behavioral observations, or analysis of written or spoken text, typically to gain an understanding of motivations or opinions.

A Quality Data Model (QDM) attribute provides specific details about a QDM data element. QDM version 5.6 includes only datatype-specific attributes, i.e., metadata, or information about each QDM datatype the electronic clinical quality measure (eCQM) developer may use in eCQM expressions to provide necessary details for calculation.

A Quality Data Model (QDM) category consists of a single clinical concept identified by a value set. A category is the highest level of definition for a QDM element. The QDM currently contains 22 categories. Some examples of categories are Medication, Procedure, Condition/Diagnosis/Problem, Communication, and Encounter.

A Quality Data Model (QDM) data element encapsulates a certain category with an associated datatype. It is a discrete unit of information used in quality measurement to describe part of the clinical care process, including a clinical entity and its context of use. It can include criteria for any relevant metadata about a clinical or administrative concept relevant to quality measurement. A QDM data element provides an unambiguous definition and enables consistent capture and use of data for quality measurement. The measure developer may define for any given measure and reuse when they require the same information for another measure. Reuse encourages standardization of quality measures and reduces the generation of additional software requirements for every new measure.

A Quality Data Model (QDM) datatype is the context in which each category is used to describe a part of the clinical care process. Examples of QDM datatypes include "Medication, Active" and "Medication, Administered" as applied to the QDM Medication category.

Quality Data Model (QDM) entities represent concepts used to specify details about the actor (or performer) of any QDM datatype. An electronic clinical quality measure can use the entities to provide further information required for an individual or organization actor to meet the measure's criteria.

A quality measure is a standard for measuring the performance and improvement of population health or of health plans, providers of services, and other clinicians in the delivery of health care services (§931). Patient Protection and Affordable Care Act of 2010, Pub. L. No. 111-148, 124 Stat. 119 (2010). https://www.govinfo.gov/content/pkg/PLAW-111publ148/pdf/PLAW-111publ148.pdf

Established by CMS, QualityNet provides health care quality improvement news, resources, data reporting tools, and applications used by measured entities and others. QualityNet, the one-stop shop for CMS quality programs, provides information on quality data reporting programs associated with the Hospital Inpatient Quality Reporting Program and other hospital inpatient quality programs, including the Hospital Outpatient Quality Reporting Program, Ambulatory Surgical Center Quality Reporting Program, Inpatient Psychiatric Facility Quality Reporting Program, Prospective Payment System-Exempt Cancer Hospital Quality Reporting Program, End-Stage Renal Disease Quality Incentive Program, and the Medicare Promoting Interoperability Program. CMS encourages interested parties, advocates, and others to register for email notifications to receive information on enhancements and new releases, timelines, process or policy modifications, and alerts about applications and initiatives. The CMS Hospital Quality Reporting program notification and discussion lists are available for signup on QualityNet.

A quantitative assessment uses numerically based methods, typically supported by statistical analyses, to show relationships between independent and dependent variables to facilitate description, prediction, and establishing of causal relationships.

A ratio is a score derived by dividing a count of one type of data by a count of another type of data. For example, the number of patients with central lines who develop infection divided by the number of central line days. The key to the definition of a ratio is the numerator is not in the denominator.

The reliability subcriterion assesses the measure to ensure it is well defined and precisely specified so measured entities can implement consistently within and across organizations and distinguish differences in performance.

A resource use measure, also called a cost and resource use measure, is a measure of health services counts (in terms of units or dollars) applied to a population or event (including diagnoses, procedures, or encounters). A resource use measure counts the frequency of use of defined health system resources. Some may further apply a dollar amount (e.g., allowable charges, paid amounts, or standardized prices) to each unit of resource use.

A respecified measure is an existing measure changed to fit the current purpose or use. This may mean changing a measure to meet the needs of a different care setting, data source, or population. It can also mean changes to the numerator, denominator, or adding specifications to fit the current use.

A sandbox allows development and testing of a software application in an isolated and controlled environment. For testing of applications such as clinical decision support, a sandbox will need to contain a sufficient amount of realistic data to mimic application functioning in the clinical system. 

The scope delineates what is included and excluded in a project. It may define specific products (also known as deliverables) or specific processes that will occur as part of a project. Scope definitions can also include descriptions of assumptions, constraints on the project, and acceptance criteria. 

Scoring is the method(s) applied to data to generate results/score. Most quality measures produce rates. However, other scoring methods include categorical value, count, continuous variable, frequency distribution, non-weighted score/composite/scale, ratio, and weighted score/composite/scales.

Semantic validation is a method of testing the validity of an electronic clinical quality measure (eCQM) whereby the eCQM developer compares the formal criteria in an eCQM to a manual computation of the eCQM from the same test database.

Sensitivity, as a statistical term, refers to the proportion of correctly identified actual positives. For example, the percentage of people with diabetes correctly identified as having diabetes. See Specificity.

Several hospital-related quality measures have an associated short name. For example, Venous Thromboembolism Prophylaxis has the associated short name VTE-1. CMS and interested parties use the short name in documentation and it is often the term preferred by measure users.

SMART-on-FHIR apps are application programming interfaces using the Substitutable Medical Applications and Reusable Technologies (SMART) platform in concert with Fast Healthcare Interoperability Resources® (FHIR®) to provide a standards-based method for authentication, authorization, and retrieval of clinical data as well as interoperable data exchange with electronic health records.

SMART objectives are a structured approach to achieving project goals by focusing on objectives that are Specific, Measurable, Achievable, Realistic, and Time-bound (SMART).

A specification is a measure's instructions addressing data elements, data sources, point of data collection, timing and frequency of data collection and reporting, specific instruments used (if appropriate), and implementation strategies.

Specificity, as a statistical term, refers to the proportion of correctly identified negatives (for example, the percentage of healthy people who are correctly identified as not having the condition). Perfect specificity would mean the measure recognizes all actual negatives. For example, recognizes all healthy people as healthy. See Sensitivity.

Users of Health Level Seven International® (HL7®) standards use a standard for trial use (STU) to provide timely compliance with regulatory or other governmental mandate and/or timely response to industry or market demand. HL7 incorporates an STU, following a suitable period for evaluation and comment, into fully balloted and accredited version of the standard. Formerly called draft standard for trial use.

A standard operating procedure is a set of fixed step-by-step instructions or steps applicable to routine operations or situations with the intent to improve efficiency, uniformity, and quality of operations.

The Submission Tool and Repository (STAR) is a measure database of quality performance measures submitted to the CMS consensus-based entity (CBE) for endorsement review. The database includes historical data for measures submitted for endorsement and maintenance to date. The CMS CBE updates the STAR database regularly as the Partnership for Quality Measurement receives submissions for endorsement review for new and maintenance measures. 

Stratification divides a population or resource services into distinct, independent groups of similar data, enabling analysis of the specific subgroups. This type of adjustment can show where disparities exist or where there is a need to expose differences in results.

A structure measure, also known as a structural measure, is a measure assessing features of a health care organization or clinician relevant to its capacity to provide health care.

A subject matter expert is an individual with specialized expertise in a specific area or field.

Synthea™ is an open-source, synthetic patient generator that models the medical history of synthetic patients. It provides realistic, but not real, patient data and associated health records covering every aspect of health care. The resulting data is free from cost, privacy, and security restrictions, enabling research with health information technology data that is otherwise legally or practically unavailable.

Browse and download pre-generated synthetic records.

A systematic review is a scientific investigation focusing on a specific question and using explicit, pre-specified scientific methods to identify, select, assess, and summarize the evidence from similar, but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data.

A tag is a keyword or term assigned to a piece of electronic information to serve as metadata and facilitate search processes.

The target/initial population refers to all events for evaluation by a specific performance measure involving patients or events who share a common set of specified characteristics within a specific measurement set to which a given measure belongs. Draw all patients/events counted (e.g., as numerator, as denominator) from the target/initial population.

A technical expert panel (TEP) is a group of experts and other interested parties who contribute guidance and thoughtful input to the measure developer or other group seeking advice and expert information from representatives from multiple interested party groups for the purpose of obtaining balanced input representing varied perspectives. Measure developers involve TEPs in every stage of the Measure Lifecycle, from conceptualization through maintenance.

The Technical Release Notes (TRNs) are resource files, which provide an overview of technical changes, e.g., logic and terminology, for each electronic clinical quality measure (eCQM) in the most recent eCQM Annual Update for CMS reporting/performance periods. Each grouping, e.g., Eligible Clinician, Hybrid Measures, has their own TRN file.

A terminology system is a set of terms representing the system of concepts of a particular field.

The Unified Medical Language System® (UMLS®) is a set of files and software bringing together health and biomedical vocabularies and standards to enable interoperability between computer systems. The UMLS integrates and distributes key terminology, classification and coding standards, and associated resources to promote creation of more effective and interoperable biomedical information systems and services, including electronic health records. Software developers can use UMLS to enhance or develop applications, such as electronic health records, classification tools, dictionaries and language translators. Use of UMLS allows end users to view codes, synonyms, and definitions used terminologies such as Common Procedural Terminology and International Classification of Diseases-10th Revision-Clinical Modification and relationships between concepts. Access requires a free UMLS License.

Unified Modeling Language (UML) is a standardized methodology-independent approach to specifying, visualizing, modeling, and documenting the structure and design requirements of business processes, such as software development.

The usability and use criteria examine the extent to which potential audiences (e.g., consumers, payors, measured entities, policymakers) are using or could use performance results for accountability and performance improvement to achieve the goal of high-quality, efficient health care for individuals or populations. 

A use case is a unique instance of sharing a specific type of information regarding patients and their health. Each use case has a specific purpose, type of data exchanged, and rules for interactions between people and systems. Examples of use cases include immunization records sent to the state government for public health reporting, and admission notifications sent to doctors and other members of a care team when one of their patients has a hospital admission.

User experience is a person's perceptions and responses resulting from the use and/or anticipated use of a product, system, or service.

A user story is a succinct, plain-language description of a desired software feature written from the viewpoint of the end-user.

The validity subcriterion assesses whether measure specifications are consistent with the measure intent and capture the most inclusive target population. There are two main types of validity, measure validity and data element validity. Measure validity is when the measure accurately represents the evaluated concept and achieves the intended purpose (to measure quality). Data element validity is the extent to which the data element or code represent the information. 
See also measure validity and data element validity.

A value set is a list of specific values, terms, and their codes, used to describe clinical and administrative concepts in quality measures. Value sets provide groupings of unique values along with a standard description or definition from one or more standard vocabularies used to describe the same clinical concept, e.g., diabetes, clinical visit, demographics, within quality measures. Examples of standard vocabularies used to support effective, interoperable health information exchange include SNOMED CT, RxNorm, and Logical Observation Identifiers Names and Codes.

The Value Set Authority Center (VSAC) is a value set authoring tool and repository. It is the source of truth for value sets for CMS's electronic clinical quality measures. The National Library of Medicine maintains the VSAC and provides access via web browsers and APIs. The VSAC provides downloadable resources for all measure releases that provide full eCQM value sets, the Binding Parameter Specification, and the complete lists of Direct Reference Codes, retired codes, and utilized code systems. The VSAC provides users with tools to search existing value sets, create new value sets, and maintain value set content consistent with current versions of the terminologies they use. Access requires a free Unified Medical Language System License. 

A value set expansion is the actual list of codes, calculated using a specified expansion profile of code system versions and any predetermined retired (legacy) codes used with implementation of the value set.

The Virtual Medical Record (vMR) is a data model for representing the data analyzed and/or produced by clinical decision support engines.

Visual Studio Code is an open-source text editor that can expand its utility by installing appropriate extensions. The ‘cqframework.cql’ extension supports highlighting, syntax checking, ad hoc evaluation, test cases, terminology for Clinical Quality Language (CQL), and Fast Healthcare Interoperability Resources® model validation. This free text editor can be used to facilitate electronic clinical quality measure evaluation and testing using the open source CQL Evaluation Engine.

The Value Set Authority Center (VSAC) Collaboration Tool provides a central site where value set authors can post value sets for collaborative discussion. Teams can share threaded discussions about the value sets, view recent value set expansions posted by other value set authors, organize their value sets by usage and by team’s workflow needs, collaborate with other value set authors to harmonize value sets, and receive activity and change notifications from VSAC. Access requires a free Unified Medical Language System (UMLS) license.

The VSAC Support Center is the Help section of the Value Set Authority Center website. It provides information on how to use the VSAC, specific information for value set authors and stewards (e.g., how to create value sets), and additional information such as release notes and how to use the VSAC Collaboration Tool, and VSAC Application Programming Interface Resources.

The Value Set Authority Center (VSAC) Sharing Value Set (SVS) Application Programming Interface (API) allows the user to retrieve a single value set's expansion, with or without its metadata and definition version, using RetrieveMultipleValueSets and RetrieveValueSet, respectively. It is based on the Integrating the Healthcare Enterprise (IHE) SVS Technical Framework, section 2.2.21 SVS integration Profile and the IHE SVS XML Schema. Access requires a free Unified Medical Language System (UMLS) license.

A workflow is a repeatable sequence of steps or tasks needed to complete a specified process. In a health care delivery setting, workflow refers to a directed series of physical and cognitive activities performed by the care delivery team or equipment/computers to assess, change, or maintain the health of a patient as part of the delivery of clinical services. 

A workflow diagram is a visual representation of the sequence of steps or tasks needed to complete a specified process.