Patient-Reported Outcomes

Abstract

Clinical studies frequently rely on patient-reported outcomes to fully evaluate the efficacy of a drug, device or treatment. This chapter differentiates between traditional and electronic methods of capturing patient-reported outcomes and discusses features of each approach. The chapter also examines the impact of regulatory requirements on patient-reported data collection.

Introduction

Certain situations necessitate information be reported by a subject rather than being objectively measured by study personnel. These types of data are known as patient-reported outcomes (PRO). PRO data give researchers the opportunity to quantify subjective experiences, which can be crucial in studies that measure, for example, symptoms, disability, emotional state, social functioning, or subjects’ perceived response to symptoms, treatments or disability. PRO data also allows for data collection outside of scheduled visits and without any investigator interpretation of data.

The processes involved with the development of a PRO questionnaire are part of a field known as psychometrics. Psychometrics can be described as a scientific discipline concerned with the theories and techniques of questionnaire construction, quantification, testing and validation. Psychometrics relies heavily on statistics, and a well-designed questionnaire will reliably measure the attribute(s) it is designed to measure.

Many clinical studies use PRO data to some degree, and for some of these studies PRO data is a primary efficacy parameter. If an investigator cannot observe, quantify or measure a variable, it may instead be reported by the subject. The best way to learn about an individual’s personal experience is to ask the individual to describe it. In some cases, PRO data can also be used for registration or economic evaluation purposes.

Although PRO data have been used in clinical studies for many years, the advent of electronic tools may improve the quality of data and ease of data collection. Electronic patient-reported outcomes (ePRO) provide potential advantages over traditional PRO collection methods, but there are circumstances where ePRO may not be the best choice for a study. Each study should be individually evaluated to determine which collection method is most appropriate.

Scope

This chapter discusses various methods of capturing both traditional (sometimes referred to as paper-based or non-electronic PRO) and electronic PRO data. The chapter also describes different types of PRO data, advantages of different data collection methods, and clinical data management (CDM) considerations when choosing a PRO or ePRO method for a study. For this chapter, the term “questionnaire” refers to any instrument or measure, including patient diaries, used to collect PRO data.

Minimum Standards

• Provide detailed instructions to subjects for completion of any PRO questionnaire, whether electronic or paper-based.

• Employ strict version control on all PRO questionnaires. Changing one word of one question, changing the order of questions or changing the instruction to subjects can invalidate the comparability of results.

• If PRO data-collection tools are administered in multiple languages, ensure all translations are linguistically consistent.

• Ensure processes are in place to assure compliance with regulatory requirements regarding protection and ownership of ePRO data (electronic source data).

• Consult with information technology personnel to ensure networked ePRO tools have the appropriate level of network security and infrastructure.

• Ensure ePRO systems are properly validated in compliance with US 21 CFR Part 11 and any other regulations specific to the location of the study.1

Best Practices

• Use PRO only for variables that cannot be directly measured. For example, do not ask a subject if their reflexes have slowed when a neurologic exam can be administered instead.

• Use standardized, validated questionnaires when possible to avoid needing to test the psychometric properties of a newly developed PRO questionnaire. For example, there should be no need to develop a questionnaire to screen for depression when many standardized questionnaires already exist.

• Document the development processes for newly developed PRO questionnaires.

• Document any study-specific modifications or revisions to PRO questionnaires. Refer to the FDA guidance to confirm whether changes require additional testing before implementation.2

• Conduct appropriate training and retraining (as necessary) with subjects to familiarize or refamiliarize them with the PRO questionnaire used. Giving subjects hands-on experience with ePRO tools may be more critical than with paper-based PROs, but adequate training should always be ensured regardless of collection method.

• Use a standard predetermined structure for collecting subject data (i.e., interview scripts, questionnaire layouts, electronic devices, telephone prompts).

• Ensure PRO questionnaires have been thoroughly psychometrically tested. Consult with a statistician for any questions about quality of psychometric testing.

• Avoid post data collection queries for missing or inconsistent data, as these data are a subjective account of the subject’s experience at a particular point in time and no additional source is available to cross- check reported data. Because queries are not generated for PRO data, the resultant database may contain inconsistencies that are not addressed, regardless of whether ePRO or traditional paper-based PRO capture methods are used.

Data Suitable for PRO Collection Methods

One approach to PRO data capture is intended to quantify the status of particular conditions or symptoms. These PRO data are often collected with questionnaires that ask questions about symptoms associated with a condition and use an established algorithm to quantify the results. This can include the state of discrete symptoms such as pain severity, or can be an assessment of the overall state of a condition such as depression or asthma. Numerous questionnaires exist employing this approach, many of which have been psychometrically tested. Some of these questionnaires may be protected by licenses, and appropriate authorizations may be required to use the questionnaires or modify them in any way, even if this modification only involves transferring the questionnaire to an electronic format or a different language.

Another type of PRO data involves a subject’s self-assessment of feelings or opinions. This can include feelings or perceptions about a condition or treatment, or a self-assessment of current or recent emotional states such as depression or anxiety. Assessments of a subject’s feelings about a treatment must oftentimes be designed for each study, as questionnaires may need to be tailored to the specific treatment.

PRO data may also involve subjects’ self-assessments of their activities, social or physical functioning. These types of assessments may ask the subject how their social or physical functioning has been over a period of time, and these assessments frequently ask about specific activities or interactions. A number of widely accepted questionnaires exist to quantify some of these types of PRO data.

Finally, PRO data may consist of a subject reporting the frequency of certain events, such as bowel movements, headaches or taking pain medication. Because these data consist of counts, little or no psychometric testing may need to be applied. These data may be used as an indicator of treatment exposure, efficacy or safety or as an indicator of study compliance.

Traditional PRO Collection Methods

The following methods are sometimes referred to as paper-based PRO, although in some cases the subject may report these outcomes verbally while a researcher records the information through an electronic medium.

• In-person or telephone interviews—In this form of PRO data collection, a researcher elicits verbal responses from a subject. The interview should be scripted and administered consistently following established guidelines. This approach reduces likelihood of a question being overlooked; however, subjects may be reluctant to share some personal information with an interviewer. Although this may be described as a paper-based PRO collection method, researchers may record subject responses on paper or electronically.

• Paper questionnaires—This is the most commonly used PRO collection method, and psychometrically tested paper questionnaires already exist for much PRO data. When subjects visit a study site for an assessment, treatment or follow-up visit, they complete paper forms designed to quantify various PRO data. Paper questionnaires may also be mailed to subjects for completion. A disadvantage of paper questionnaires is that subjects may not always answer all questions. When questionnaires are completed at a study site, a cursory review may confirm whether a question was intentionally left blank. Mailed questionnaires, however, are especially prone to missing responses.

• Paper-based diary—Diaries are meant to assess subjective information when subjects are going about their normal lives. Subjects may be asked to enter information at certain intervals (e.g., daily or hourly) or may be asked to record when certain events occur, such as an asthma attack or insomnia. Although diaries can capture a wide range of subjective information, they are susceptible to subjects filling in information both backward and forward in time. This is sometimes known as “parking lot compliance,” where a subject completes pages meant to cover a range of time, but instead completes all the pages at once, for example, in the parking lot of the doctor’s office before a visit.

Characteristics of Traditional PRO

Traditional PRO collection methods sometimes hold advantages over ePRO. A side-by-side comparison of traditional PRO characteristics and ePRO characteristics can be found in Table 1. Some of the characteristics of traditional PRO capture include:

• Fewer startup resources (e.g., hardware, software, technical support) are typically needed for traditional PRO capture.

• Minimal setup time is usually required for traditional PRO capture.

• Site personnel do not need to train subjects in use of the capture instrument, because most people are familiar with paper questionnaires.

• Because some subject populations may be more comfortable with paper than electronics, there could be a potential for bias in subject selection if ePRO is used.

• Traditional PRO capture methods are not as susceptible to the impact of technology failures such as battery depletion, device malfunctions, busy telephones, Web server crashes, or ineffectual help desk support.

• Traditional PRO may be associated with compliance issues due to lack of subject surveillance.

ePRO Collection Methods

Technologic advances have enabled researchers to utilize electronic tools to capture PRO data. In most cases these tools provide greater overall efficiency or improved data quality, but they may not be the most appropriate solution in all cases. The study team should evaluate each study on a case-by-case basis to determine the best approach. The following tools are in common use today, but additional PRO data capture methods may emerge in the future.

• Handheld devices—Although hardware costs typically make study setup more expensive than other methods of capturing PRO data, personal digital assistants (PDAs) or other handheld electronic recording devices allow comprehensive capture of PRO data. These devices combine portability, ease of use, and the ability to capture a wide range of PRO data. The convenience and portability of these devices promotes reporting information in real time rather than asking subjects to remember a prior period of time and then accurately report the requested information. Many devices can also be programmed to provide subjects with reminders of scheduled times to record information.

• Web-enabled reporting—This approach allows subjects to fill out questionnaires or diaries from a computer connected to the Internet, which allows comprehensive data capture, but lacks the portability of a handheld device. In addition to allowing a full range of PRO data collection, Web- enabled reporting can be relatively inexpensive to set up.

• Interactive Voice Response (IVR) systems—This method utilizes automated interactive telephone systems to capture PRO data. These systems typically are not as ideally suited for collecting as wide a range of data as handheld devices or Web-based questionnaires. For example, phone systems do not have an acceptable way to complete a Visual Analog Scale (VAS) or indicate the precise location of pain. Subjects also may not always have access to a phone when calls are meant to be made. Although this is classified as an ePRO capture method, subjects may perceive this as more similar to some traditional methods.

Characteristics of ePRO

When used effectively, ePRO can provide some advantages over traditional PRO data collection methods. A side-by-side comparison of traditional PRO characteristics and ePRO characteristics can be found in Table 1. Some of the characteristics of ePRO include:

• Greater data accuracy may be associated with ePRO because improved surveillance may promote more timely data entry.3 More timely data entry may translate into more accurate or more complete subject reporting. With paper-based diaries, there could be a tendency for subjects to fill in a week’s worth of data in the parking lot of the site before a visit.4, 5 Using ePRO allows all data entry to be date and time stamped, helping to ensure the subject is entering information based on their recall at that specific point in time. This helps to avoid recall bias, which can be a confounding factor in many of the subjective measures that are captured by PRO instruments.

• Potential for improved subject compliance is provided through some ePRO systems’ features such as automatic reminders, as well as the convenience of portable ePRO devices that allow more flexibility of when and where data are entered.

• Potential for fewer errors exists with ePRO because of the lack of ambiguous or unusable data due to illegible handwriting that may be associated with traditional PRO capture methods. The potential for fewer errors can also be facilitated by various front-end edit checks such as minimum/maximum values, time windows, response choice rules, etc.

• Reduced burden and increased convenience for subjects can be provided by ePRO capture methods, in part because ePRO offers question branching, which can allow for fewer, but more targeted questions.

• Quicker sponsor access to subject-reported data can be provided by ePRO capture methods, enabling proactive real-time study management. This could be helpful for studies with adaptive design. For example, if a study design includes study endpoints or decisions that are contingent upon PRO data, ePRO can provide advantages by making these data available more readily.

• Electronic integration with the clinical database is accommodated by many ePRO capture methods.

• More confidential collection of sensitive data may be achieved by ePRO capture methods.

• Some ePRO tools offer the opportunity to integrate interactive training for subjects.

Choosing a Pro Method

ePRO is not always readily available for some locations or populations. 

CDM personnel should carefully evaluate which PRO or ePRO capture methods will provide the best results for a study. The following factors should be considered when determining whether to use ePRO or traditional PRO.

• Complex or lengthy questionnaires may not be suitable for some capture methods, such as IVR systems.
• Studies of long duration may be subject to changes in technology used for ePRO data collection.
• The degree of psychometric testing applied to the mode of administration should be considered.2, 6 For example, an established questionnaire will typically have more scientific validity than a structured interview.
• Clinical subject population and demographics may affect the suitability of some methods.
• An ePRO system’s compliance with regulatory requirements should be thoroughly examined.
• Connectivity and data transmission abilities relative to the operating locale must be taken into account. For example, Web-enabled reporting may be a poor PRO capture method in a rural area with limited Internet access.
• The technological capabilities and quality standards of available vendors should be considered
• If a study design involves study endpoints or decisions that are contingent upon PRO data, ePRO can provide advantages by making these data available more readily.

The following table presents some considerations of both traditional PRO and ePRO such that the two collection strategies can more easily be compared and contrasted.

Table 1. Comparison of Paper-Based PRO and ePRO Methods

Costs Considerations

One of the most common arguments against using ePRO is that it is in many cases more expensive to set up than a traditional PRO approach. Although ePRO costs may be more expensive during study setup, the possible savings ePRO may offer become more apparent when viewed in the context of an entire study. A careful assessment of the following areas will help determine the most cost-effective manner to collect PRO data for a study.

• General considerations (fixed vs. recurring costs, amortization of fixed costs over multiple studies, need to acquire in-house expertise vs. outsourcing)

• Resources available to the organization at the beginning of a project

• Hardware and software expenses

• Personnel additions or reductions (less personnel may be needed for data entry and data cleaning depending on the collection method)

• Training requirements and costs

• Help desk or support needs for subjects and investigator sites

• Adequate infrastructure for hosting ePRO data

• Adequate disaster recovery plans to ensure continuous access to ePRO data

• Additional tools that may be used with ePRO (such as IVR systems, handheld devices or a Web page)

• Programming and integration costs must be considered for ePRO

• Validation requirements when going from paper to ePRO (e.g., when you already have a fully validated paper-based questionnaire)

Regulatory Considerations

Use of PRO or ePRO is subject to the same regulatory oversight as any other tool used in a clinical study. CDM personnel should take measures to ensure regulatory requirements are met for sponsors, vendors and investigators in regard to record keeping, maintenance and access. All ePRO tools should be fully compliant with 21 CFR Part 11, including a comprehensive audit trail preserved for every step of the data collection and handling processes.

Guidance for Industry: Patient Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (DRAFT) provides guidance for the selection or creation of a PRO questionnaire or test. It should be noted that this draft guidance also states, “If a patient diary or some other form of unsupervised data entry is used, the FDA plans to review the protocol to determine what measures are taken to ensure that patients make entries according to the study design and not, for example, just before a clinic visit when their reports will be collected.”2

For widely accepted questionnaires, ample information usually exists in the scientific literature to support the psychometric properties of the instrument. However, for modified or newly created instruments, “The FDA generally intends to review a PRO instrument for: reliability, validity, ability to detect change, and interpretability (e.g., minimum important differences).”

Recommended Standard Operating Procedures

• CRF Design
• CRF Completion Guidelines
• Data Review
• ePRO System Validation
• Vendor Management