3 Determining Treatment Effectiveness

STUDY DESIGNS AND STRENGTH OF INFERENCE

Questions of treatment effectiveness are fundamentally questions about cause-and-effect relationships. If an effective treatment is applied, some detectable improvement in a patient’s condition should occur. If the treatment is not applied, no improvement occurs or the patient gets worse. If the treatment is applied in higher doses or more frequently (at least up to a point), the improvement should be greater or occur sooner. Because there may be other causes of improvement beside the treatment in question, the improvement must be shown in multiple patients in multiple settings and in circumstances where as many other possible causal factors can be ruled out.

Assessments of the efficacy of specific treatments typically start with some evidence of biological plausibility. Basic laboratory studies or other kinds of knowledge that do not involve direct tests of a treatment in live human patients may suggest that a treatment or class of treatments should work. No matter how compelling the arguments for plausibility, though, plausibility per se is not evidence for treatment efficacy.

The randomized controlled trial (RCT) is the most reliable methodology for assessing the efficacy of treatments in medicine. In such a trial a defined group of study patients is assigned to either receive the treatment or not, or to receive different doses of the treatment, through a formal process of randomization. A coin flip is the simplest example of a random process. In a study with two “arms” (e.g., treatment or no treatment), each eligible patient would receive whatever a coin flip indicated—heads for treatment and tails for no treatment. In a large number of patients, any clinical or demographic factors such as age, height, weight, illness history, other illnesses, or any other unknown factor that might affect the results of the treatment would be equivalent in the two groups. These will all be eliminated, then, as plausible competing explanations for any observed difference in outcome between the two groups.

Randomized trials typically include other features that increase the strength of the conclusions about cause-and-effect relationships between the treatment and the outcome of interest. Some patients may be excluded from the study because they have conditions that make it impossible to evaluate outcomes or gather data (e.g., extremely elderly patients may be excluded from a study of a cancer treatment because too many of them would die of other conditions before the end of a five-year follow-up period). The study of the efficacy of a drug may include lab tests that measure the level of the drug in the bloodstream. This is done to ensure that the patients assigned to the treatment group actually received the drug while the patients randomized to “no treatment” did not take it on their own. A study may include near-term clinical measures of benefit (e.g., reduction in blood pressure or cholesterol level) as well as long-term objective measures of benefit (e.g., remissions of tumors, mortality) or long-term subjective measures of benefit (e.g., self-reported pain or functional status levels).

Even though an RCT provides strong evidence for or against the efficacy of a given treatment being tested, no one study is ever so perfect that the results cannot be challenged. A study may show an absence of an effect of a drug, for example, because the dose chosen for study was too low. The patients in a particular study may be unique in some way that makes them not representative of all patients to whom the treatment might be given in the future. For example, if all study subjects are middle-age white men, it is not clear whether the treatment would work in the same way for older or younger Asian women. Results that appear significant in a near-term follow-up may change with longer follow-up (e.g., a treatment shrinks tumors dramatically for two months, but the cancers recur and the patients die after 18 months). The most powerful evidence of treatment efficacy comes from the cumulative, consistent results of several RCTs, preferably in different patient populations and in different settings, and with extensive follow-up periods.

Other kinds of studies (i.e., quasi-experimental designs) can provide evidence of treatment efficacy, too. In situations where it is technically or ethically impossible to run concurrent control groups, a series of “off/on” periods of treatment in a single group of patients can be studied. In these studies treatment is administered to a single group of patients and then taken away. Evidence of efficacy is provided if the benefit is consistently seen when treatment is given and the benefit disappears when treatment is not given. This is a specific example of a before-after study design without controls. A single round of off/on provides very weak evidence for effectiveness unless results are unusual and dramatic, because many other things occurring at the same time as the treatment may have caused the result. Being able to repeat the effect over and over again strengthens the argument for the treatment, rather than something else, being the cause of the effect.

One could also use a cohort study. In this study design a large number of patients who receive a treatment are followed over time to observe a possible benefit and are compared to those who did not receive the treatment. It may offer strong evidence of treatment effectiveness if the group studied is particularly large so that other possible causes of an effect may be evaluated through statistical analysis, or if the result is unusually strong and/or consistent in the large group. For example, one might observe a lower rate of heart attacks in a large group of men taking an aspirin every day over a 10-year period compared to similar men who did not take it. One might challenge the results, though, and ask whether the men taking aspirin became more health conscious in general and also lost weight, drank less, quit smoking, or did something else that was actually the cause of the reduced rate of heart attacks. Being able to go into the database and find the effect in a subset of men who did not lose weight or quit smoking or drink less would offer an answer to the challenge.

Another possible study design is a case-control study. In this study design, patients are assigned to study groups based on the results of treatment rather than the treatment itself. One might, for example, identify some patients who survived one year after a heart attack and other patients who died within a year. If most or all of the patients who survived received a certain kind of treatment and few or none of those who died received the treatment, the treatment might have improved the odds of survival. In this kind of study and in the cohort study, there is no direct control over who receives what treatment, so there may be competing explanations for the effect. Perhaps only healthier, stronger patients received the treatment, with the possibility that the survival benefit was related to their health and strength rather than the treatment. Because this design leaves open many competing explanations, it is less powerful than an RCT for assessing treatment effectiveness and is usually followed by an RCT to confirm findings.

Still weaker evidence for treatment effectiveness comes from uncontrolled clinical observations or anecdotes. These types of studies may involve small numbers of patients, treatments that are not well defined or that vary from patient to patient, variable periods of follow-up, unstated or varying rules for selecting patients for study, or outcome measures of unknown validity (e.g., patient says he “feels better”). They may not be studies at all in the sense that there is any organized effort to answer a scientific question about a treatment. They may simply be the collected reports of the experiences of patients who share some basic characteristics of illness and treatment. Although these kinds of studies can appear in major professional journals because they do provide some evidence of potential treatment effectiveness, they are generally not considered to provide strong evidence because of the many possible competing explanations for the observed effect that they cannot rule out.

In summary, there are a number of study designs that can provide varying levels of evidence of treatment efficacy. They include (from strongest to weakest):

• Multiple well-designed randomized controlled trials
• Single well-designed RCT or multiple small RCTs
• Cohort study, particularly one with multiple “on/off” features
• Case-control study
• Series of clinical observations or anecdotes

EFFICACY VERSUS EFFECTIVENESS

The discussion above has used the term “efficacy” to refer to the typical change in a patient’s (or group of patients’) health status brought about by a given treatment. There is an important distinction, however, between two similar-sounding terms:

• Efficacy is the benefit produced by a given treatment in tightly controlled, perhaps artificial, study conditions in which patients are carefully selected and may be more frequently observed, tested, and monitored than is typically the case in routine clinical practice.
• Effectiveness is the benefit produced by the given treatment in dayto-day clinical practice, in unselected patient populations that do not receive extra tests, education, or visits because of participation in a study.

With this distinction in mind, most RCTs would be properly labeled efficacy studies rather than effectiveness studies. The results, then, reflect treatment efficacy rather than treatment effectiveness. Treatment effectiveness would typically be established after one or more RCTs, when the treatment came to be widely used in a variety of clinical settings in diverse patient populations.

It is possible to imagine an “effectiveness RCT” that would combine the design features of the RCT with a set of sampling and analytical features that would permit direct extrapolation of findings to routine clinical practice (Roper et al. 1988). An effectiveness RCT could be viewed as a hybrid that combines the real-world features of effectiveness studies with some of the study design features typically found in efficacy studies. In an effectiveness RCT, one would have relatively light (if any) patient exclusion criteria, so that the patients in the trial would be as similar as possible to those to whom the results would be generalized. The study would be run (to the extent possible) in a range of treatment settings rather than in a single academic medical center context. The treatment would be provided by the same kinds of providers (e.g., community physicians or nurses) who would provide the treatment in nonstudy settings. There would not be an elaborate data collection infrastructure (e.g., extra lab test or imaging studies) that would create a different “information environment” for treating clinicians and patients than the one that would be found in real-world treatment settings. Analysis would be done on an “intention to treat” basis. The study would have random assignment of patients to treatment arms and would have one or more control groups (e.g., placebo controls, waiting list controls, different dose or regimen controls, or other controls that would make sense for the question being asked).

If we adhere to this terminology, we will find that there is very little formal evidence of treatment effectiveness for most treatments for medical problems in Gulf War veterans because relatively few true effectiveness studies have been done on any medical condition.

OUTCOMES RESEARCH

In an influential 1988 article, Paul Ellwood coined the term “outcomes management” and challenged the medical community to formally assess patient outcomes, not only in the context of specific research projects but also in the context of routine clinical care. His concept of outcome not only included the directly measurable outcomes of clinical trials like objective tumor shrinkage or mortality, but also a range of self-reported patient outcomes like functional health status, pain, ability to work, and overall quality of life. Part of his concept included the building of large data repositories in which clinicians would build a “collected clinical experience” of the baseline characteristics, treatments, and outcomes of thousands and thousands of patients. With this database available, he argued, future clinicians could enter the characteristics of a patient with a given problem and almost immediately obtain data on the relative effectiveness of several treatment choices in patients like the one in question.

There are a number of outcomes research projects that represent the closest approximation to a study of true treatment effectiveness (Magid et al. 2000). The National Registry of Myocardial Infarction is one example; the APACHE III and the Spine Surgery Consortium for Outcomes Research are others. The American Medical Group Association has been the sponsor of a large number of related outcomes projects that have been carried out in the past 10 years.

In the scheme of study designs described above, these studies may perhaps best be categorized as cohort studies, although the cohort is defined in terms of the clinical condition rather than by the receipt of a specific treatment. They do not typically have the tight control of patient eligibility criteria and the random assignment of treatments that are the hallmarks of the RCT. It is presumed that the effects of possible alternative causal factors can be identified and controlled through appropriate statistical techniques, and this is made possible by large sample sizes and the collection of a wide range of relevant variables.

The committee is not aware of any ongoing outcomes management or outcomes research projects involving either a specific cohort of Gulf War veterans or a specific clinical condition of particular concern to Gulf War veterans. There clearly is an opportunity, though, in both VA and DoD settings to organize such a project since some of the functions of identifying a cohort of patients and gathering baseline data have already been completed through their registry programs.

META-ANALYSIS

We have stated that the strongest form of evidence for treatment effectiveness is that of several well-designed RCTs whose results Mahjong Ways are consistent. What about the situation, though, in which study results are not fully consistent or where there are multiple studies of differing sizes and degrees of design rigor? The technique of meta-analysis was developed in an attempt to fit this situation.

In a meta-analysis the results of multiple studies are combined to yield an overall cross-study estimate of treatment effectiveness (DerSimonian and Laird 1986). The key characteristics of a meta-analysis include:

• explicit criteria for deciding which studies are relevant and are to be included;
• explicit criteria for reviewing published or unpublished literature and choosing candidate studies for the analysis;
• explicit criteria for grading studies according to rigor of design and execution and resulting strength of evidence;
• explicit criteria for assigning “weights” to individual studies that reflect the strength of evidence in each one; and
• a statistical method for aggregating the results of different studies that may have widely varying sample sizes, definitions of study end-points, follow-up intervals, and statistical tests of effect.

In principle, the results of a good meta-analysis should provide the closest possible approximation to “definitive evidence” of treatment effectiveness, since they are based on a formal, well-defined integration of the results of multiple studies conducted in different populations in different settings. Just as is the case with other study designs, however, meta-analysis does not always live up to its promise. It is necessary to carefuly examine the criteria and methods used in each meta-analysis to determine the quality of evidence provided by its conclusions.