Metrics in Clinical Data Management

Abstract

A wide range of measurements, commonly referred to as “metrics,” are essential to evaluate the progress and outcomes of a clinical study. This chapter considers various metrics used in clinical data management, as well as the process of selecting metrics that are related to the goals and objectives of an organization. The chapter discusses the importance of standardizing metrics for a project and across projects, and gives suggestions to help ensure metrics are provided in a timely fashion with adequate contextual information to be understood and effectively used to measure, monitor performance and improve efficiencies.

Introduction

The term “metric” simply refers to a measurement. In clinical data management, metrics can quantitatively and qualitatively assess whether or not a process or individual or group performance is efficient and effective, as well as indicate whether the factor being measured has or will have an expected level of quality. Metrics can be used at various intervals throughout a study to ascertain if processes are working as planned. When a process has been completed, well-designed metrics can help indicate if goals were achieved with the expected level of quality.

This chapter provides information on metrics that are particularly relevant to clinical data management (CDM) personnel. There are no regulatory mandates regarding specific metrics; however, metrics can assist in detecting potential regulatory issues, for example by measuring compliance with SOPs. The effective use of metrics also helps an organization evaluate and improve quality and productivity. This chapter is intended to provide helpful suggestions and considerations for CDM personnel involved with establishing a metrics program within a department or company.

Scope

For the purposes of this chapter, the term “metrics” primarily refers to specific data management process-related measurements assessed during the course of a study, but may also refer to the data generated by these measurements. Roles and responsibilities vary between organizations, and some of the topics discussed in this chapter may be the responsibility of different departments in different organizations. Regardless of role assignment, CDM personnel should be aware of the processes discussed in this chapter and how they impact their roles as data managers.

Minimum Standards

• Ensure CDM metrics are aligned with key performance indicators (KPI) (milestones, deliverables, timelines and other quantitative measurements) to meet the organizational needs and goals.

• Ensure that all metrics are clearly defined, quantifiable, documented and approved.

• Communicate approved metrics to relevant personnel and stakeholders within and across projects.

• Ensure adequate and appropriate resources (hardware, software, personnel, etc.) are made available to accurately and thoroughly measure and report metrics.

• Ensure the personnel responsible for defining, quantifying, documenting and communicating metrics have the proper training and relevant skills and competencies.

• Ensure all personnel and stakeholders are adequately trained regarding metrics definition and their relevance to process and project performance.

• Perform quality assurance on data used to determine the metrics, to ensure that the metrics are based on accurate and timely data.

• Establish and document corrective action to be taken if planned or actual metrics do not align with goals and objectives.

Best Practices

• Include all stakeholders (e.g., project managers, clinical leads, data managers, management, etc.) in the development of metrics specifications.

• Ensure all stakeholders (e.g., project managers, contractors, clinicians, data managers, and management) understand and agree with the definition of the measurements and the parameters used to provide each metric before implementing use of the metric.

• Align metrics with project team/organizational goals as well as industry standards and contractual agreements, when and where appropriate.

• Standardize the definitions of metrics by using consistent terminology and parameters across projects and the organization.

• Agree upon well-defined metrics at the onset of a project, and use those metrics to evaluate performance during all stages of the project.

• Select a set of key metrics that apply to all projects. Use these metrics as the basis for comparison of process performance across all projects.

• Consider the aspects of cost, quantity, quality, timeliness and performance when deciding which metrics to implement.

• Identify metrics that will indicate progress to targets and also provide insight into historical performance.

• Ensure that the effort needed to collect and report a metric is appropriately offset by the benefit. Where possible, implement automated collection of data for metrics, and strive to use existing primary data (e.g., audit trails, tracking systems) to collect metrics.

• Ensure the tools used to collect and report metrics are thoroughly validated, and are 21 CFR Part 11 compliant where applicable.

• Establish benchmarks of expected performance based on pooling of similar data.

• Ensure metrics findings are visible to relevant stakeholders via a reporting plan (charts, dashboards, etc.) followed by a feedback loop and rigorous action plan through root cause analysis (RCA) and corrective action/preventive action (CAPA).

• Document the process for collecting, reporting, and communicating metrics.

• Evaluate metrics collection and reporting processes frequently (for both internal and outsourced activities)

• Determine if metrics need revision, or if other metrics should be added or eliminated, based on changes in technology or process landscape.

Identifying Metrics

An organization’s use of a set of key and relevant metrics will facilitate achievement of predetermined goals. Although agreement on certain metrics is obtained by the overall company or department, individual departments or project teams may need to maintain additional metrics to assess the progress toward the goals of their respective department or team.

Metrics should be based on goals and objectives set by an organization, and ideally, organizations and departments should strive to identify a set of metrics to use across all projects. Identifying the specific metrics that fit the needs of all involved parties is often difficult. Most goals and objectives set by groups or organizations revolve around the interdependent areas of quantity cost, time, quality, and performance, as shown later in this chapter in Table 1.

• Quantity - Quantity measurements are straightforward and objective, and are therefore among the easier metrics to quantify.

• Time - When measuring time, one of the most important considerations is defining the exact start and stop points and the unit of measure (e.g., business days, calendar days, or resource hours). Time measurements ensure that chronology of milestones is maintained. Organizations may follow a risk-based approach in adhering to the timelines over other metrics.

• Cost - Although costs are not typically a CDM responsibility, CDM may supply metrics that are used for cost analyses.

• Quality - Quality is the most important metric to be considered in CDM. Quality metrics may measure the quality of processes and deliverables and can be quantified in different ways. For more information about data quality, see the GCDMP chapters entitled “Measuring Data Quality” and “Assuring Data Quality.”

• Performance - Metrics intended to quantify performance are typically made up of some combination of measures of quantity, time, cost, and quality. Therefore, performance can also be assessed in terms of one or more of these measures in relation to another measure, such as performance over time, or performance compared to cost. Performance should typically be measured at multiple levels (for example, site, study, project etc.)

When considering a set of key metrics, an organization should design the metrics to allow for their application across projects, regardless of the project-specific process or technology used. This approach allows for an assessment of each project in comparison to similar projects. It also allows for an evaluation of processes that may be redesigned to take advantage of a new technology.

Two examples applicable for clinical studies using either paper-based data collection or electronic data capture (EDC) are: measurement of the number of queries per data field for incoming data as opposed to the number of queries per page and measurement of the time from subject visit to data entered in the database.

Clinical studies are often evaluated within the realm of strategic (i.e., organizational) and tactical (i.e., operational) objectives. Metrics assessments are generally based on the relationship between two or more (e.g., quantity over time, or quality of quantity) of the five core criteria of quantity, time, cost, quality, and performance.

One should be cautioned that focusing too much on one criterion may adversely affect another. For example, focusing too strongly on quality may impact study timelines, similarly focusing too strongly on study timelines may negatively impact quality. All of the above-mentioned criteria should be balanced to some degree in the metrics used by an organization.

Regardless of the measurement, or why a measure exists, a well-designed metric should be

• relevant - answers critical business questions

• enduring - is of lasting relevance

• robust - is not subject to manipulation or variation due to process changes

• valid - measures what it implies to measure accurately

• specific - is clear and consistent

• actionable - can drive decisions

• practical - is measured in a timely fashion without a significant drain on resources.1

The effort needed to collect and report a metric should be offset by the potential benefit. If a metric has no benefit, it should not be collected just because doing so is easy and inexpensive. Cost, quality, and performance metrics may be difficult to quantify, whereas metrics dealing with quantities and times are often much easier to collect. The metrics that are collected and reported should be able to answer questions that have been predefined to measure the success or failure of a project or process.

Linking Metrics with Organizational Goals

A hierarchical relationship exists between the objectives of an organization, a department, and an individual project or clinical study. An organization may have strategic objectives that include achieving a certain level of quality in its product while achieving a particular profit margin at the end of the fiscal year. Each functional group within an organization, such as CDM, sets tactical goals and objectives to ensure quality while using resources efficiently. A particular project manager or project team may have budget and time constraints, yet be expected to deliver a quality end product.

Each functional group must develop its own objectives and metrics within the context of the organization’s objectives. However, cross-functional input should be solicited to ensure consistent interpretation of the metrics. The existence of these hierarchical objectives and concurrent timelines drives the need for consistency in the definition and utilization of metrics.

Linking Metrics with Project Goals and Deliverables

Overall project goals and objectives must be considered when metrics are selected and evaluated. A set of metrics that only addresses some, but not all, of the five core criteria will provide only a partial assessment of overall project performance. If one metric is met, it does not imply that the others are achieved. For example, even if milestones are achieved on schedule, they may have required additional resources.

In addition to overall project goals, metrics should also be considered in relation to specific deliverables. For example, if the database lock is scheduled by a certain date, metrics that may indicate the possibility of delays should be carefully examined and communicated. This ensures that all stakeholders have realistic expectations of when the database lock will actually occur.

Even when the same set of metrics is used across projects, they may be prioritized differently for each project. For example, cost containment may be assigned a higher priority in an early phase exploratory study, while data quality may be prioritized in a phase III pivotal trial.

Identifying Users

To optimize the effectiveness and efficiency of metrics, the users of each metric should be clearly identified. Each metric should be linked with documentation of who collects the metric, who reports the metric, and who is responsible for initiating any actions that may be taken based on the metric. If a metric is to be used for evaluating progress toward goals, all such stakeholders should be identified and documented.

Metrics should be shared with all stakeholders participating in a project when applicable, including CROs and vendors. Decisions should be made early in the project planning stages concerning which metrics will be collected, who will collect the metrics, how and when the metrics will be disseminated (e.g., with a common Web site or visualization tool, such as a dashboard, one month after the first patient signs the consent form, etc.).

Metrics results should be communicated to relevant stakeholders clearly and within prescribed timeframes, enabling needed corrective actions to be made in a timely manner.

Evaluating Metrics from Various Sources

Obtaining metrics can be difficult when the parameters required for measurement are found in multiple databases. Even if all of a study’s clinical data reside in a single database, data comprising project metrics may originate from a study database, a project tracking system, a CDMS (clinical data management system), or a system outside CDM altogether. This issue is further compounded when certain complementary metrics, such as the project budget and the status of various CDM processes, are not available for equivalent time frames. However, metrics can be synchronized with other relevant information if they are collected in a timely manner.

Automated data generation for metrics that can be shared electronically across various systems, will lower the chance of errors and the effort needed for re- entering the data. The use of technologies such as Web portals, clinical trial dashboards and visualization tools is a viable option for reviewing metrics data allowing proactive control of study progress. All such tools used in the clinical data management environment must be validated to ensure accuracy.

These tools may have the capability to aggregate real-time study data into intuitive views, eliminate the need to integrate databases or re-enter data, and allow for views of complementary data within the same time frame.

Metrics in Different Types of Studies

EDC systems offer the capability to have clinical data and queries available sooner (in real time) than in paper-based studies. Study or subject status indicators such as subject enrollment or visit completion may also be available within the EDC system. The quality and timeliness of metrics improves substantially when they are collected electronically.

In paper-based studies, CDM metrics can be generated electronically only after data are entered into the database or CDMS. Information regarding subject enrollment, visit completion, and other such status indicators can be difficult to obtain in a timely fashion. Teams often rely on each site to report this information (e.g. using paper enrollment logs) and then subsequently re- enter the information into a project-management or project-tracking database.

Metrics Common to EDC and Paper-based Studies

Many metrics common to EDC and paper-based studies relate to overall performance of the project, team, or organization. Because metrics measuring organizational or group performance are not contingent upon the data collection modality used, they are also usually independent of any CDMS or database software. Although there are some exceptions, most well-designed metrics are not dependent on a particular data collection strategy or software package.

Metrics Unique to Paper-based Studies

Data entry is one area in which metrics for paper-based studies may be created. An example is the percentage of data entered relative to the number of completed CRFs received. Another example is performance metrics for data entry personnel (number of forms/patients entered per day, per employee). Paper-based studies will also have metrics related to data clarification forms used for query resolution, which are not needed in EDC studies due to the capability of generating queries electronically.

Some metrics used in paper-based studies may have a different meaning when used in EDC studies. For example, data entry percentage may also be measured in studies using EDC, although in that case it is an indication of site performance.

Metrics Unique to EDC Studies

EDC-specific metrics are often directly associated with the EDC system. Examples include the percent of EDC system downtime or the average number and severity of EDC help desk calls. Another class of unique EDC metrics are those that would be prohibitively expensive to measure in a paper- based study, such as the number of modules pending PI review and signature. For more information about metrics in studies using EDC, see the GCDMP chapter entitled “Electronic Data Capture—Concepts and Study Start-up.”

Importance of Metrics Standardization

Because metrics may be shared between various functional groups or stakeholders, metrics should be based on standard definitions. The need for standardized definitions is amplified if metrics are used for comparisons across studies, projects, or organizations (e.g., benchmarking projects). Communication between various groups using a metric is also enhanced by the use of standard definitions.

For example, “time to database lock” is one of the most frequently cited metrics used in clinical studies. However, this metric may be defined differently within different organizations. Depending on an organization’s definition of this metric, completion of database lock may be considered to occur:

• when data are “frozen” and a sponsor accepts data transferred from their CRO (e.g., the database or transferred datasets),

• after a QA audit is accepted and it is deemed permissible to break blinding of the study,

• multiple times, depending upon SOPs and whether or not a company allows for database “unlocking” to make changes to the database after it was originally locked.

Likewise, the starting point for this metric may be defined by different organizations as any one or more of the following criteria:

• the last subject completes the last visit (LPLV),

• the last data from the last subject visit are recorded on a paper CRF or entered into an EDC system,

• the last CRF is received by the group performing data entry,

• the data cleaning activity is deemed completed (i.e., generation of last query in database).

• the last query or discrepancy is resolved.

Due to various interpretations of the metric “time to database lock,” all parties could potentially be working in different directions based on their presumption of when database lock occurs and what activities take place at that point. Without a standard definition of this metric, the goal may not be identified or achieved in an efficient and effective fashion. To ensure clarity and efficiency, all functions affected by a metric should be involved in the definition of the metric and made aware of the interpretation of the metric that is to be followed.

If the starting point for “time to database lock” is the date the last subject completes the last visit, the CRA or monitoring group should work with CDM to develop and agree upon definitions and the process used to achieve this milestone. As for the end point, if it is defined as the point that blinding of the study is broken, appropriate representatives (e.g. biostatistics, CDM and personnel responsible for randomization code storage) should work together to understand their respective roles in this process. The data management plan (or other applicable documentation) should be kept current to reflect any decisions that are made regarding metrics to be collected and their definitions.

Like other areas of clinical data management where standards are evolving, there is an initiative to develop industry-wide standards by the not-for-profit Metrics Champion Consortium (MCC). Comprised of representatives from biotechnology, pharmaceutical, medical device and service provider organizations, the mission of MCC is to develop performance metrics within the biotechnology and pharmaceutical industry.

One of MCC’s initiatives focuses on clinical trial metrics where more than forty performance metrics have thus far (as of March 2011) been defined along with standard formulas and calculations used for reporting. Paired with standardized definitions and standard formulas for measuring each metric, all parties can stay informed of the criteria for measurement and the results being achieved not only within an individual study, but across studies that also use the identical metric definitions and formulas.

Figure one shows an example schematic of performance metrics within a clinical study and indicates when specific metrics may be used and the focus of that metric (for example, to evaluate quality or efficiency).

Figure 1. Example Schematic of Performance Metrics within a Clinical Study.

Legend:

Context and Attributes of Metrics

The context in which a metric will be applied should be determined prior to reporting the metric. Each metric’s data source(s), data extraction date, and reporting window should be included with each report. Each metric should also be grouped according to its attributes, which can be described as characteristics of a metric that help stakeholders understand the underlying causes for performance variances.

Some attributes that may be used for grouping include:

• therapeutic area,

• indication,

• study phase,

• data collection mode (e.g., EDC, paper, imaging),

• study design,

• size or complexity factors (e.g., number of sites, number of subjects, number of procedures), or

• resourcing model (e.g., CRO, contractors, in-house staff, etc.).

Categorizing and summarizing metrics according to their attributes can result in more clear and concise metrics reporting, and minimize the potential for making invalid assessments and generalizations.

Defining Time Points for Standardized Metrics Collection

To provide maximum benefit, metrics reports should be available for review as soon as possible. Project and department managers frequently need to gather status information for an ongoing study, including information such as enrollment rates, the number of open queries, or the types of queries that occur most frequently on CRF data. The greatest opportunity to take corrective action occurs when information is timely. The earlier a problem is detected, the sooner it can be addressed. Although details may vary between organizations and studies, Table 1 presents some metrics commonly used during different periods of a study. Although the table groups these metrics by the five core criteria and by three study periods (startup, conduct and closeout), some of these metrics may be applicable to multiple criteria or time periods.

Table 1. Examples of Common Study Metrics

Action Plans: The Feedback Loop

Ultimately, the desired outcome of using metrics is obtained through well- planned and executed processes that include interim assessments and feedback loops. An organization should carefully design the procedures that collect the metrics needed to assess whether a goal has been reached. However, the organization should also carefully design procedures describing the actions that may be taken based on the results of collected metrics.

Metrics reports are useful for both interim and final assessments of a project, therefore these reports should be run at agreed-upon times during and at the end of the project. Reports should summarize the metrics collected, and should include an assessment of results against goals or objectives. Metrics reports may also provide commentary about the results, which should include reasons for positive performance and plans for corrective action to improve performance.

Useful reports for the analysis of metrics include trend analyses, statistical techniques, summary tables, flagging of outliers, identifying unanticipated trends in the data, plots showing incoming data and query rates, and listings of values, such as changes from baseline values.3 Ideally, metrics should be categorized according to their ability to assist in comparing a project’s outcome to the outcomes of other projects inside or outside the organization.

Using Metrics to Improve Organizational Efficiency and Effectiveness

Comparing metrics from different projects and studies can help improve the overall efficiency and effectiveness of an organization. If a particular process functioned more effectively and efficiently in a specific project, the organization can try to determine what factors made the process more efficient in that specific project and then try to apply those same factors to other projects. By using metrics to identify areas of strength or weakness within individual projects, an organization can apply lessons learned to projects in the future, thus improving the overall effectiveness and efficiency of the entire organization.

One of the means of ensuring visibility and transparency of metrics across all parties (sponsor, clinical research organization, and vendor) is by creating service level agreements (SLAs) and operational level agreements (OLAs) for those metrics that form the key performance indicators. Routinely reviewing KPIs in governance meetings (strategic and operational) provides an indication of the health of the project and may identify areas needing corrective and preventive actions (CAPA).

Using Metrics to Improve Timeline Efficiencies

Metrics can be used early in a study to identify areas where timeline efficiencies might be improved. For example, if a particular site is not entering data or resolving queries in a similar timeframe as other sites or within the expected timeframe, the root cause can be identified and, if warranted, corrective and preventive actions can be initiated, such as retraining relevant site staff. If particular milestones are not being reached as expected across an entire study, processes and data collection tools can be reevaluated to determine if adjustments could potentially improve timeline efficiencies.

Using Metrics to Improve Operational Efficiencies

Frequently, operational efficiencies can also be improved by initiating corrective actions based on metrics reports. As with timeline efficiencies, identified operational inefficiencies at a particular site (e.g., delay with uploading data from ePRO) can often be improved by retraining relevant site staff. If metrics identify processes that are not working as efficiently as intended across an entire project or study, relevant processes and tools can be carefully examined to determine the most effective corrective actions needed to improve operational performance and efficiency.

Metrics Documentation

The data management plan (DMP) is a tool that can be used to document decisions about the use of metrics for a project (e.g., metrics definitions, the means of collecting metrics, the means of communicating metrics). However, some organizations may choose to document metrics separately from the DMP. Regardless of where they are documented, the metrics used for a project should be defined at the planning and initiation stages of the project.

All key metrics reports and other documents relevant across projects should be referenced in the project documentation, as well as all project assumptions and assertions for establishing particular metrics. If new terms are used or new stakeholders or vendors are involved with a project, establishing and maintaining a project dictionary or glossary may be helpful.

Recommended Standard Operating Procedures

• Definitions and Use of Performance Metrics
• Validation and Testing of Metrics Collection Tools
• Vendor Management