The cumulative knowledge base about the effectiveness of treatments of human disease is prodigious. Nonetheless, the use of many of the most common therapeutic interventions in clinical practice is not supported by evidence derived from clinical trials. Clinicians often rely on their experience with similar patients, supplemented by published case series, to estimate the likelihood of relevant health outcomes with different treatments. This less-than-rigorous approach produces different opinions about treatment effectiveness, which lead to wide variations in clinical practices for ostensibly similar patients. Because of lessstringent regulatory control, the knowledge gaps are generally greater with devices and surgical procedures than with drugs. For example, transurethral prostatectomy represented the standard of care for benign prostatic hyperplasia for 40 years before publication of a clinical trial.

Even when randomized trials are performed, uncertainty remains about the effectiveness of treatment for a specific patient. This in part reflects the methods used in clinical trials to measure the isolated effects of the treatment being studied. Explicit inclusion and exclusion criteria are often applied to ensure that the study population is homogeneous to minimize the effect of patient-specific variables on outcomes. Patients with severe disease or important comorbid conditions are often excluded. Others may be excluded because of age or sex. Patients who meet criteria are then randomized to one of two or more carefully defined treatment strategies. Barriers are erected to discourage patients from switching from one treatment to another, and other steps are taken to preserve the integrity of the treatments.

The course of patients in well-conducted randomized trials is carefully monitored, with equal attention given to patients regardless of treatment, and their outcomes are determined by means of explicit, predefined measures. Ideally, both patients and those who measure outcomes are unaware of the treatment assignment, even if this requires the use of placebos or sham procedures. Otherwise, expectations may have a real effect on perceived treatment outcomes and be interpreted as specific effects of treatment.

All of these steps serve to protect the validity of the trial as a test of the hypothesis that the treatments compared in the trial have different effects on outcome. However, these same steps often limit the applicability or generalizability of the study findings to patients for whom the study treatment might be considered. The clinician must be concerned both about the internal validity of the trial as a test of the hypothesis about treatment
effectiveness and its external validity, that is, the extent to which results are applicable to the patients seen in practice, who may be quite different from patients in the trial.

Treatments may also vary from one setting to another. This is especially true for surgical procedures and for “complex interventions,” which are composed of a number of components, which may act both independently and interdependently. Regarding surgery, many studies have suggested that the volume of particular procedures performed in a hospital and/or by particular surgeons is inversely related to mortality rate. One study indicated that these differences in risk can be quite significant clinically for complex procedures, including pancreatectomy, esophagectomy, and pneumonectomy. Other characteristics of hospitals and of particular surgeons have been studied less well.

In the case of complex interventions, the effectiveness can vary significantly with the context of the environment in which it is designed and implemented. This is often the case when the intervention involves comanagement of the disease by the clinician and the patient or a caregiver.

Concerns are also raised when there are multiple goals, either stated or unstated, for a particular trial. All trials involve potential trade-offs in the welfare of trial participants and those future patients who would benefit from the highest achievable scientific validity of the study. Different perspectives exist on appropriate times to stop trials early because of apparent benefit, thereby forgoing the more precise estimates of benefit and harm.

Some trials are designed to compare a new agent and an established agent and prove not a greater benefit, but rather noninferiority to the active-control agent. Such noninferiority may justify a change to the new agent when it has other advantages related to cost, convenience, or safety, but in an age of direct-to-consumer promotion of new drugs through highly selective presentation of evidence, there may be reason for concern.

A similar concern about the construction of trials that may allow for misleading conclusions derives from the use of “composite endpoints,” made up of one or more of several events of interest, as the primary outcome of a trial. The advantage of using a composite endpoint is an increased event rate and increased power of the trial for a given number of patients. However, some events, such as all-cause or disease-specific death, may be uniformly important to all patients, whereas other events, such as new onset of angina, might be less important, and this might be variably so among patients. Trials using composite endpoints can be misleading if the number of more important events is small and the magnitude of effect varies across elements.

All too frequently, individual trials may not be large enough to provide definitive answers to clinical questions. Increasingly, data from multiple trials are being combined in meta-analyses to allow careful systematic interpretation of the evidence. With the proliferation of randomized trials in the last decade, systematic reviews, or meta-analyses, have become the gold standard for treatment effectiveness. However, systematic reviews have their own problems in design, conduct, and interpretation. For example, to combine results of trials that measure the same outcome on different scales (e.g., one of several depression scales or quality-of-life measures), the author of a systematic review must calculate the standardized mean difference for each trial. Although the statistical process is straightforward, it has been shown that errors in data abstraction capable of significantly altering results are not uncommon.

There are other challenges in combining the results of different trials. There may be evident clinical heterogeneity of the patient populations, the intervention, or both. Interventions designed to support patients in decision making or self-management often vary from one trial to another and from one setting to another. In other cases, the populations and interventions may seem similar, but the results of trials may not be consistent with one another. This statistical heterogeneity can be seen in a “forest plot” of trial results or determined with statistical tests for heterogeneity that produce a high P-value when heterogeneity is low and a low P-value when heterogeneity is high. If heterogeneity is high, it may be explained by unappreciated clinical differences, methodologic problems, or publication bias. The meta-analysis can proceed using a random-effects model rather than a fixed-effects model. This means that because of the heterogeneity, the effects of treatment are assumed to vary around an overall average treatment effect rather than the same common fixed effect. However, in situations of high heterogeneity, the clinician interpreting the systematic review may want to step back and ask whether it made sense to combine the trials that were included.

Despite the problems that sometimes arise in combining studies, systematic reviews provide a great resource for the clinician who strives to make the most of clinical knowledge in the care of patients. A prodigious amount of work has been done under a number of different auspices; most notable are those of the Cochrane Collaboration (see later discussion, under Making the Most of Available Resources). That work is necessarily ongoing. Even the most robust of systematic reviews is always at risk of going out of date. One study found a mean duration of “survival” without the need for updating of 5.5 years for a cohort of 100 systematic reviews. However, 23% were in need of revision within 2 years and 15% within 1 year. There is also a need to make the results of systematic reviews more accessible to clinicians and policy makers. Standardized formats for different audiences have been proposed and are increasingly being used.