Assuring Data Quality

Owned by Hieu Dao
Last updated: May 19, 2017
20 min read

Abstract

High quality clinical research data provide the basis for conclusions regarding the safety and efficacy of a medical treatment. This chapter discusses how the terminology and methodology for assuring quality, already well established in other industries, can be applied successfully to clinical research. General principles of quality systems and quality assurance in clinical data management are discussed. The key differences between quality assurance and quality control are presented and the roles of standardization, standard operating procedures, and auditing are reviewed.

Introduction

Before discussing methods of assuring data quality, one must determine exactly what is meant by terms such as “quality,” “quality control” (QC) and “quality assurance” (QA). The American Society for Quality (ASQ) provides the following definitions for these terms.

  • Quality - This is a subjective term for which each person or sector has its own definition. In technical usage, quality can have two meanings: 1. the characteristics of a product or service that bear on its ability to satisfy stated or implied needs; 2. a product or service free of deficiencies. According to some experts on quality, such as Joseph M. Juran, quality means “fitness for use,” and according to Philip B. Crosby, it means “conformance to requirements.”

  • Quality control - This term refers to the operational techniques and activities used to fulfill requirements for quality.

  • Quality assurance - This consists of all the planned and systematic activities implemented within the quality system that can be demonstrated to provide confidence that a product or service will fulfill requirements for quality.

In clinical data management, QA may be thought of as an overall management plan to ensure the integrity of data (the “system”), while QC may be thought of as a series of measurements used to assess the quality of the data (the “tools”). The terms QA and QC have been used in an imprecise manner in many industries including clinical research, and the ASQ and the American National Standards Institute (ANSI) both provide explanatory notes to that effect with their definitions of these terms.3,4 The Institute of Medicine definition is often used for data quality within the context of clinical data management (CDM), and states quality data “are those that support the same conclusions as error free data.”

A key aspect to remember about clinical research data quality is that it may be composed of numerous attributes.6 For clinical research data, attributes of quality may include accuracy, consistency, timeliness, consumability, currency, completeness, relevance, granularity, unambiguity, precision and attribution. Clinical research data quality may therefore refer to a dataset that accurately represents data points collected from subjects, has acceptable completeness, is defined sufficiently for use, is current, is attributable, and contains relevant data at the appropriate level of precision to answer the study’s primary hypotheses.

Quality assurance refers to all of the planned actions and systems implemented to impart confidence that a study will culminate with a quality dataset. Within this context, quality control refers to specific activities and techniques employed within the QA system to achieve the goal of finishing the study with a quality dataset. The most common approach to assuring quality is through a quality management system, which is the means by which an organization is controlled with respect to quality.8

Although the ultimate goal of CDM personnel is to complete a study with a quality dataset, proper principles and practices must be employed throughout the course of a study to ultimately ensure quality. If a study’s design, protocol or case report forms (CRFs) are of insufficient quality, the study is unlikely to accurately provide answers to its hypotheses. Lack of quality processes in any part of a clinical study can lead to results that are distorted, missing or inaccurate.

Scope

This chapter emphasizes the infrastructures and practices that those managing clinical research data should use to ensure data quality. Although quality measurement methods are a necessary part of a plan to obtain quality data, a larger emphasis should be placed on error prevention, both in organizational infrastructure and early in the design stages of each protocol. For information about identifying and quantifying errors in clinical research data, see the Good Clinical Data Management Practices (GCDMP) chapter entitled “Measuring Data Quality.”

Many of the tasks described in this chapter may be joint responsibilities between different groups, just as many different groups may be involved in the implementation of various tasks. However, in all cases clinical data managers need to be conscious of whether or not these tasks have in fact been performed in a satisfactory manner.

Minimum Standards

  • Design and maintain data-handling processes according to the organization’s documented quality system.

  • Attempt to collect only data that are essential for interpretation of study results and that are required by the protocol.

  • Provide sufficient information in data-processing documentation to reproduce final analyses from source data.

  • Assure data quality for all studies, whether submitted for regulatory review or not (e.g., marketing studies, observational studies or for publication-only studies).

  • Ensure data quality is appropriate for study analyses according to parameters laid out in a statistical analysis plan, if one exists. Appropriate levels of data quality for analyses should always be determined by an experienced statistician.

  • Use company-standardized data collection and handling processes.

Best Practices

  • Have an organizational quality policy that is strongly supported by upper management, understood by all staff, and supported by operational procedures.

  • Create and maintain documentation of all roles and responsibilities involved in managing a clinical study.

  • Use industry-standardized data collection and handling processes.

  • Use well-documented processes for data collection and handling.

  • Minimize the number of data-processing steps in order to minimize potential sources of error.

  • Focus on error prevention with QA and focus on process monitoring with QC. The final product (database or software) of the clinical study should not be the focus of QA or QC.

  • Ensure data quality audits assess compliance of procedures to regulations, compliance of practices to written documentation, conformance of data to source documentation, and conformance of data to written procedures.

  • Apply data QC to each step of data management processes.

  • Ensure all data management personnel are trained on and knowledgeable of the organization’s quality policy.

Quality Systems

A quality system encompasses the organizational structure, responsibilities, procedures, processes, and resources that are necessary to implement quality management.9 This approach was standardized by the International Organization for Standardization (ISO) and is applicable across many industries, including clinical research. A quality system approach advocates an infrastructure that provides the flexibility to account for study differences in a controlled and consistent manner. Although not mandated for all clinical studies, a quality system approach has been adopted by the FDA in medical device regulations.10

Every study should establish an appropriate minimum level of quality, which should be determined through planned analyses specified in the protocol or statistical analysis plan. The assessment of data quality needs should address the study’s purpose, characteristics and complexity.5 A key concept of the quality system approach is that the structure, format, content, and method of presentation of documented procedures are contingent upon the needs of the organization.9 Most organizations involved with clinical research already have some components of a quality system in place, for example, policies and procedures.

Within the context of CDM, a quality system should assure the following fundamentals:

  • Written procedures and associated documentation should enable the clinical database to be reproduced from the site’s source documentation.

  • Written procedures must be followed.

  • Data are consistently of sufficient quality to “support conclusions identical to those drawn from error free data.”

ISO Quality Systems

The ISO provides the ISO 9000 series of standards to assist organizations with creating and maintaining quality systems.9 The ISO quality management system describes a process-based approach in which organizations establish the infrastructure needed to control quality of their product sufficiently to meet customers’ needs consistently.9 To meet ISO quality management system infrastructure requirements, an “...approach to developing and implementing a quality management system consists of several steps including the following:

    1. determining the needs and expectations of customers and other interested parties;

    2. establishing the quality policy and quality objectives of the organization;

    3. determining the processes and responsibilities necessary to attain the quality objectives;

    4. determining and providing the resources necessary to attain the quality objectives;

    5. establishing methods to measure the effectiveness and efficiency of each process;

    6. applying these measures to determine the effectiveness and efficiency of each process;

    7. determining means of preventing nonconformities and eliminating their causes;

    8. establishing and applying a process for continual improvement of the quality management system.” 9

Implementing a quality system starts with identifying processes that are required to produce a product. In CDM, these are processes for which most organizations already have standard operating procedures (SOPs). A quality system, however, goes beyond SOP documentation and includes confirmation that a methodology is effective, resources are available, and measurement and monitoring are sufficiently rigorous (i.e., a control cycle for those processes such as periodic process audits). The ISO standard provides specific documentation requirements as well as necessary roles and responsibilities.

Components of a CDM Quality System

The components of a CDM quality system must take into consideration the practices and elements of a quality system infrastructure. The ISO quality system requirements can be translated into the following areas for CDM.

  • Defined processes necessitate that all operations performed by CDM are identified and defined. The starting point is an inventory of processes for which the department or group is responsible. The quality system standard also requires specification of the sequence of processes, as well as interactions between processes. The quality system should be consistently applied to all departments of an organization, because CDM is but one component of the clinical research process—data are also collected at sites, verified by monitors and analyzed by statisticians. For departmental implementations of the standard, document the interface points of CDM processes with processes from other departments. These interface points can be documented in SOPs for data management processes.
  • Position descriptions list and describe the functions of specific jobs or titles. Position descriptions should accurately and thoroughly describe the requirements of a position, including responsibilities, tasks and education. Position descriptions serve as the basis for candidate selection, training, performance evaluations, and promotions. Each individual involved with a study should have a position description that accurately describes the work they regularly perform.
  • Training is described in more depth in the “Training” chapter of the GCDMP. Both the ISO standard and FDA regulations require that individuals have documented training for their job tasks. For each work process in which an individual participates, training should be provided and documented. Organizations often create a training matrix listing each position and required training for each position. All job description tasks should be linked to SOPs and be adequately represented in the training matrix, although there is no regulatory requirement to provide these links.
  • Management oversight is a good practice. Even if a quality system has documented work processes, job tasks, and training, factors such as comprehension, quality, judgment and consistency can vary. Many CDM tasks require review of an individual’s work, as well as an opportunity for the individual to receive constructive feedback. Although some review may be appropriately conducted by a peer, management oversight should also occur above the level of the individual. For example, departmental management should receive summary status reports of progress and QC activities. At an even higher level, management has oversight responsibility to assure the quality management system consistently produces acceptable quality.
  • Process control refers to the capability of a process to consistently produce a particular result. Although process control can also be considered part of management oversight, it is important enough to be described separately. Management is responsible for designing and maintaining processes that produce consistent results. For CDM, consistent results may include acceptable database error rates, data timeliness, minimal errors in database programming, and meeting milestone deadlines. Rather than establish separate measures and controls for each process, process control should be determined by global organizational goals. High-level assessment may be sufficient, but detailed measures on certain processes may be helpful to identify issues early.

A quality system approach is most powerful when employed by an entire organization, covering the entire clinical research process. Although a single department can achieve high performance in isolation, only local optimization will be achieved, which may not fully align with organizational goals.

Quality System Documentation

Quality Policy

An organization’s quality policy is the highest level of a quality system. Specified by top management, the quality policy communicates and documents an organization’s overall intentions and direction with respect to quality. The quality policy should detail various levels of the organization’s quality system, such as management review procedures, the quality manual and the quality plan.8

An organization should have a written quality policy, and top-level management should demonstrate commitment to the quality policy by supporting the organization’s infrastructure with adequate resources. Off-line QC activities, such as quality engineering, quality planning, and procedures applicable to each study, will be enhanced by this infrastructure and facilitate error prevention.

Although an organization-wide quality policy as the overarching directive is best practice, if a quality policy does not exist, data management should rely on department-specific documents such as the Data Management Plan, SOPs, study-specific procedures and a study’s protocol to establish quality within the department. Because a quality product will never be achieved with only one department adhering to a quality system, it is important for data management to elevate the need for a corporate quality policy to upper management and to inform upper management of the working parameters that data management will apply in the absence of a corporate quality directive.

Quality Manual and Plans

A quality manual is defined by ISO 9000 as a “document specifying the quality management system of an organization.”8 An organization should have a written quality manual that defines the quality practices, resources, and activities relevant to the data-handling services of the organization. Most organizations already implement portions of a quality manual as SOPs, but a quality manual is much broader. A quality manual should describe not only processes, but also training, management oversight, positions, and process control.

Quality manuals and quality plans must be flexible enough to address differences in various studies. For highly standardized organizations, information that would otherwise be part of a study-specific or project- specific plan may be included in an organization’s quality system documentation (e.g., the quality manual, audit procedures, and SOPs). In these circumstances, the plan should reference these quality system documents and detail how the documents ensure data quality for each study. Quality plans may be designed to apply to one specific study or to all studies for which the organization takes full or partial responsibility. The organization’s quality plan or manual should also be subject to version and change control.

Role of SOPs

As process definitions, SOPs are a large component of (and can be specified in) the quality manual. Organizations should have a documented process for creating, reviewing, and version control of SOPs. To easily identify time periods when an SOP should be used, effective dates should be assigned to each published version. Although they do not have to be archived with each study, SOPs should be archived according to documented organizational procedures and be available should a study be audited years after closing. Planned deviations from SOPs should receive the same level of review and approval as the SOPs from which they are deviating.

The level of standardization within an organization helps determine the level of detail that should be present in the organization’s SOPs. For example, an organization with standard CRF modules, database structure, and monitoring procedures may employ detailed SOPs and thus, require less study-specific documentation.

Each GCDMP chapter recommends a corresponding set of SOPs. For those needing to create SOPs, the Society for Clinical Data Management’s (SCDM) European sister organization, the Association for Clinical Data Management (ACDM), has published Guidelines for Writing Standard Operating Procedures.

Study-Specific Procedures

A quality manual should account for the existence of study-specific documentation. Each study may have unique data-handling needs due to variations in sample size, visit schedule, type of data collected, amount of data collected, and method of data collection. Organizations should clearly document study-specific procedures to ensure the analysis database is reproducible from source documents. Study-specific procedures are also often known as data-handling plans, data management plans, data-handling protocols, and data quality management plans. Such documentation should provide supporting details to SOPs and may have a lower level of review and approval within the organization.

The ACDM has also published ACDM Guidelines to Facilitate Production of a Data Handling Protocol (DHP guidelines).12 These guidelines provide an outline and list of items to be covered in an organization’s study-specific procedures. Organizations may customize the content of the data-handling protocol, adjusting the level of detail to correspond to the level of detail present in their SOPs. The DHP guidelines are an excellent reference for defining and developing organizational study-specific procedures. Such references only provide a framework, however, and the content should be specific to the organization. For more information, see the GCDMP chapter entitled “Data Management Plan.”

Creating a Quality System

Structuring a CDM Quality System

The structure of a quality system should be designed by organizational leadership to provide consistency between studies and departments. Careful consideration should be given to what processes should remain consistent across studies and departments. The organizational QA group (if one exists) and organizational leadership will likely play an active role in establishing the appropriate level of consistency across studies and departments.

Once an organizational quality system has been designed, each department can then create and document department-specific components within the organizational structure. Although the organizational structure for a quality system is a top-down exercise and requires specialized knowledge, many of the departmental components (e.g., SOPs, training and process control) are best designed with participation from departmental staff. For example, CDM personnel will be able to suggest information that is consistent enough across studies to reside in SOPs, as opposed to information that is more suitable in study-specific documentation.

Although the level of departmental freedom to customize quality system components may vary, each department will likely have the five key components of a quality system—defined processes, position descriptions, training, management oversight, and process control.

Quality Assurance in the CDM Function

Quality assurance is the set of activities that ensures procedures are in place and effective in producing a quality product. ICH E6 defines quality assurance as, “All those planned and systematic actions that are established to ensure that the trial is performed and the data are generated, documented (recorded), and reported in compliance with GCP [Good Clinical Practice] and the applicable regulatory requirement(s).”13 In clinical research, QA includes the administration and support of SOPs and documentation. In many cases, QA also assesses the compliance of policies, products, and work processes with regulatory standards.

An organization’s written procedures should describe the approach taken to assure data are reliable and processed correctly at each stage of data handling.13 Specific tools and quantitative techniques are necessary to ensure study data meet required levels of quality at every point where data are manipulated. Process monitoring, auditing, sampling, and error rate calculation are essential processes for quantifying data quality and assessing the potential impact of data quality on a study’s conclusions. These tools and techniques should be included in the organization’s quality documentation.

Incorporating Risk-Based Assessment

Because of the time and resources needed to obtain completely clean and error-free data, a risk-based approach to QA may be adopted. Most studies do not require error-free data, but rather, data of sufficient quality to support the same conclusions as error-free data. Random data-entry errors and data that fail established edit checks may have little or no effect on conclusions drawn from statistical analyses. A QA goal in a risk-based approach would be to identify and evaluate systemic patterns of errors. Systemic errors may be considered to be non-random, for example, errors introduced through a programming fault or site-specific errors resulting from misunderstanding the protocol or CRF completion instructions. If identified, these systemic errors are typically found late in a study’s lifetime when corrective action is not as effective. Because of the number of data points in most studies, evaluating a systemic error on each data point may be an overwhelming task. A risk-based approach may be used to identify categories of data (e.g., adverse events, efficacy data, safety data) that have the highest risk levels for each study and then clean those data thoroughly.14 Risk-based practices may also include identifying higher risk studies and more stringent procedures that apply to them.

Incorporating Standards

The clinical research industry’s interest in standardization has grown in recent years. Organizations such as ISO and the Clinical Data Interchange Standards Consortium (CDISC) have published standards to provide uniform terms and structures for data collection, data storage, data transfers, and regulatory submissions. Standardization has the potential to shorten timelines, reduce costs and increase data quality.

Clinical research processes associated with data collection and handling can be error-prone and complex, potentially involving many steps. An error rate is associated with every step where data are transcribed, transferred, or otherwise manipulated. Subsequent steps can increase or decrease that error rate. Standard data collection and handling processes can be designed to limit the number of manipulations and transfers, thus reducing the potential for errors.

Regardless of its level of complexity, a standard process will become more familiar to users. Sources of error become well known and are more easily recognized and quantified, reducing unexpected errors and complications. Standardization also discourages the addition of unnecessary steps to a process. Using standard processes enables an organization to fully characterize the performance of processes and implement controlled and evaluated improvements. Successful standardization efforts can also allow the flexibility needed to address and document study-specific processes.

Opportunities for standardization may vary from organization to organization. For example, a large pharmaceutical company has more potential for standardization than does a contract research organization (CRO). For more information about standards used within clinical research, see the GCDMP chapter entitled “Data Management Standards in Clinical Research.”

Maintaining a Quality System

Once a quality system has been created, an organization’s leadership should encourage proactive maintenance of the quality system. Corporate policies often predefine the methods by which maintenance is performed. Whatever methodology is employed at a corporate level, it should not preclude employees from critiquing processes or proposing more effective and efficient practices.

CDM Quality Control

ICH E6 defines quality control as “the operational techniques and activities undertaken within the quality assurance system to verify that the requirements for quality of the trial-related activities have been fulfilled.”13 Data quality is the result of all of the planning, execution, and analysis of a clinical study. Each step in the clinical research process should be designed to ensure the necessary level of data quality is maintained throughout the study.

ICH E6 section 5.1.3 states that every stage of data handling should have QC applied to ensure data are reliable and processed correctly.13A QC step is required for each process or step in which data are transcribed, transferred, updated or otherwise saved to a new medium. When data quality does not meet predefined acceptance criteria, appropriate corrective action should be initiated.

In clinical research, data quality is typically quantified through error rate calculations. To be useful for comparing the quality of different databases, error rates must use the same scale and precision (e.g., using errors per 10,000 fields consistently rather than some combination of errors per keystroke, errors per patient or errors per record). Error rates must also measure the same components of the process and use a standard method for counting errors and fields inspected. Ideally, all error rates would represent the same sources of error and count errors in the same manner. For more information about error rate calculation, see the GCDMP chapter entitled “Measuring Data Quality.”

Error prevention, detection and monitoring activities should be described in an organization’s written procedures and documented as evidence of ongoing QC. To maximize error prevention, QC activities should occur at the earliest feasible point in a process and should assess process control and provide quantitative measures of data quality.

Some examples of QC procedures include:

  • Double data entry

  • Programmatic data range and consistency checks

  • Regular evaluation of error rates to assess process control

  • Manager or peer review of CDM deliverables (listings review, queries issued, query closing, coding)

Ongoing Process Control in CDM

Once a quality system is created and all processes are in place, personnel working within the quality system must adhere to the system for it to be effective. Management must provide oversight of process control for the quality system and ensure processes of the quality system are followed as intended, so that each function results in a quality product. For example, well- documented procedures do no good if over time, compliance decreases. Process control provided by CDM leadership helps ensure workflows and proper levels of quality are maintained.

Process control includes inspecting periodic samples of data, usually at regular intervals, and taking corrective action on the process when inspection results indicate a trend, an out-of-control process, or consistently poor quality.

Compared to the cascade effect of a design error, a process error only has an additive effect on the downstream data quality. However, each manipulation point that an incorrect data point passes through will have to be reworked to correct the error. A process that is operating in a state of control will not only meet the requirements of ICH E6 section 5.1.3, but will also reduce reworking, data cleaning, and inspection costs.

Review and Revision

Because organizations always experience change, a quality system must be able to accommodate changes. Once a quality system has been created, it should also be reviewed on a regular basis. The review may use a predetermined corporate methodology or be an ad hoc review of quality system components. Either way, if changes need to be made to the original quality system components, these changes must be reviewed and approved. Once changes have been made, all relevant personnel should be retrained on new quality system components to ensure proper implementation of the quality system.

Auditing a Quality System

The word “auditing” is described by the ASQ as a systematic and independent examination to determine whether quality activities and related results comply with planned arrangements, and whether these arrangements are implemented effectively and are suitable to achieve objectives.3

In a context more specific to clinical research, the word “audit” is defined by ICH E6 as:

A systematic and independent examination of trial-related activities and documents to determine whether the evaluated trial-related activities were conducted, and the data were recorded, analyzed and accurately reported according to the protocol, sponsor’s standard operating procedures (SOPs), Good Clinical Practice (GCP), and the applicable regulatory requirement(s).13

To be qualified to audit CDM, one should be knowledgeable of auditing methodology, CDM functions, computer programming fundamentals, and industry regulations. An auditor’s training and experience should be sufficient to thoroughly and accurately assess compliance of CDM procedures with good clinical practice. Audits of CDM functions should be performed often enough to ensure CDM processes and QC procedures effectively produce reliable and reproducible data for analysis and regulatory review.

A comprehensive audit of CDM evaluates the entire CDM quality system. The following three levels should be examined in a CDM audit.

  • Written CDM procedures should be compliant with regulatory requirements and should specify process steps and decision points required for handling and processing clinical data, including instructions for manual reviews, data-entry conventions, and data clarification procedures. Written procedures should be specific enough to enable the clinical database to be reproduced using source documentation. To determine the level of compliance with regulatory requirements, an auditor compares CDM procedures with current regulations.

  • Documented compliance of the CDM organization or department to its written policy should exist, consisting of objective evidence that the written data-handling procedures were followed. This evidence can include a database audit trail, signed and dated checklists, signed data clarification forms from a site, or interviews with CDM personnel.

  • Objective evidence should exist to indicate that CDM processes result in quantifiably high-quality, reliable clinical data for analysis and regulatory review. Several steps are required to obtain objective evidence that CDM processes produce reliable clinical data for analysis and regulatory review. The first step is quantifying the quality of clinical data, which is usually represented by an error rate. Additional objective evidence may include data demonstrating that an organization’s data-handling process is operating in a state of control. Another important type of evidence is an assessment of the potential impact of the error rate on interpretations of data and conclusions that are ultimately derived from the data. This type of assessment may be carried out by departments outside of CDM, but the results provide CDM with information that may ultimately improve CDM processes.

Other Considerations for Quality Systems

Different types of studies require different considerations in relation to QA.

Considerations for Electronic Data Capture (EDC)

For studies using EDC systems, data are available very soon after initial data collection, including audit trails, electronic signatures and query information. Review of real-time “live” data allow errors to be identified earlier in the study, as well as enabling faster subsequent corrective actions. Studies using EDC also differ in regard to source document verification (SDV). Because in some studies the EDC system can be used to capture the original recording of data (the source), studies using EDC may have fewer source documents available for SDV than would be found with a paper-based study.

Considerations for Regulated vs. Nonregulated Studies

Although regulated clinical studies undergo the additional scrutiny of regulatory authorities, data quality is critical in all clinical studies. The clinical protocol and analysis plans should drive the quality of any clinical study, whether regulated or not.

One of the primary differences between regulated and nonregulated studies is the level of risk associated with the study. Due to the differences in risk, the processes employed and the degree of QC may vary between the two. For example, nonregulated observational studies would not need as thorough and as frequent audits as a regulated study.

Considerations for External Data Sources

Vendors supplying data to be included in clinical study databases and analyses should have quality systems in place. The recipient of the data must ascertain, usually through a vendor-qualification audit, if the vendor’s quality system is acceptable and will maintain the integrity of the clinical study databases.

A study protocol will determine what external data will be transferred into a clinical study database. This requires that CDM be aware that data is expected and communicate with the data provider to negotiate the details of data transfers. If laboratory data is being handled by a central lab, communication will be on a one-to-one basis. If lab data is being handled by local labs, communication may be on a one-to-many basis and may be more complex.

The receipt of external data should be handled procedurally according to quality system components, SOPs, and study-specific requirements in the data management plan.

For more information concerning data quality from external data sources, please see the GCDMP chapters entitled “Laboratory Data Handling,” “External Data Transfers” and “Vendor Selection and Management.”

Recommended Standard Operating Procedures

  • Development and Maintenance of Standard Operating Procedures
  • Development of Planned Deviations from Standard Operating Procedures
  • Development and Maintenance of Study-specific Procedures
  • Quality Assurance Audits