At first glance, it might seem that you can discover whether a treatment is successful quite simply: just try it and see. However, a close analysis of the subject reveals that it's much harder to identify effective treatments than one might think. Decades of investigation have led scientists gradually to the conclusion that there's only one truly trustworthy source of information on whether a medical therapy really works: the double-blind, placebo-controlled study. The reasons behind this conclusion are complicated, and run counter to almost everybody's intuition. In this article, we explore this crucial topic in depth.

For a hint why double-blind studies are so important, consider the following examples: in medical trials of drugs used to treat the symptoms of menopause, many of the participants were given a fake treatment (placebo) without being informed that it was fake.1The combined results of multiple studies showed that women given placebo experienced a 51% reduction in hot flashes! Similarly, in a large study of men with prostate enlargement, participants given placebo therapy showed significantly improved symptoms and maintained at least some improvement for a full two years.2

Effects like these can be highly misleading to both physicians and patients. Suppose, for example, a physician prescribes a new drug for menopausal symptoms or prostate enlargement, and his or her patients report wonderful improvements. Does this indicate that the drug is effective? Not at all. As we know from the results described in the previous paragraph, many patients will report improvement no matter what they are taking. Thus, a drug can seem to be effective even if it doesn't possess any healing powers beyond the power of suggestion.

For a particularly dramatic example of this phenomenon, consider what happened when orthopedic surgeon Bruce Moseley, team physician for the Houston Rockets, decided he needed to properly evaluate the efficacy of an operation commonly used to treat the pain caused by arthritic knees. This surgery involves scraping away rough areas in the knee's cartilage. It is widely believed to be effective, and as many as 400,000 such surgeries are performed each year.

Mosely decided to see if the surgery really worked. He conducted a study in which five patients were given the real surgery and five were given fake surgery consisting of little incisions over the knee. He then followed his patients for two years.

The results were amazing. Interviews showed that pain and swelling were reduced just as much in the placebo group as in the group that received the real surgery. Four out of the five participants who experienced the fake surgery said it was so helpful they'd gladly recommend it to a friend. Glowing testimonials, in other words, mean nothing.

A follow-up trial of 180 individuals confirmed these results,6 and this surgical approach is on its way to well-deserved oblivion. However, if these properly designed trials had not been undertaken, surgeons might have continued to scrape arthritic knees. No doubt, there are other ineffective surgeries that pass for effective, as well as ineffective herbs, supplements, and alternative therapies, as well.

The double-blind, placebo-controlled trial is the best way to eliminate such misleading results. Such trials are the foundation of modern evidence-based medicine, and they are the foundation of the information in the Natural & Alternative Treatments database, as well.

In the following discussion, we'll begin by exploring the many factors that can deceive medical researchers. We'll follow that with an explanation of how the double-blind study design solves these problems. After that, we'll analyze the many difficulties involved in performing a meaningful double-blind study and properly interpreting the results. Finally, we'll look at other forms of scientific evidence and explain their limitations.

At least twelve factors tend to confound (confuse) the results of studies.

Observer Bias

First, researchers tend to observe what they expect to observe, a confounding factor known as observer bias. One placebo-controlled study evaluated a new treatment for multiple sclerosis.7 The researchers behind this study added an interesting twist: while most of physicians assigned to evaluate the participants for improvement were blinded, a few were not blinded, and they knew which participants were receiving placebo. As it happened, the treatment proved to be no more effective than placebo. However, the unblinded physicians managed to “observe” a significant difference in outcome between patients on placebo and those getting the treatment under study. In other words, they imagined they saw improvement where they expected to see it. No doubt this happens frequently in the daily life of a practicing physician, who is never blinded. For this reason the professional opinions of practicing doctors are far less reliable than the outcomes of double-blind, placebo-controlled studies.

Rosenthal Effect

Not only do observers’ expectations influence their own observations, they can also cause study participants to behave in the way the observers expect. This is the Rosenthal effect, and it is startlingly powerful. In one famous set of experiments, graduate students were given several photographs and told to show them to their subjects.3 The subjects were supposed to rate their impression of the people in the photos on a scale whose extremes were "big success in life" and "utter failure in life." (The photos were selected from magazines and were intended to show rather normal people.)

Next, half the graduate student experimenters were informed by their professors that their subjects would rate most of the people in the photos as failures. The rest of the graduate students were led to expect their subjects to rate the photos as showing only successful people.

Almost invariably, subjects gave precisely the ratings experimenters expected. This is particularly amazing because the graduate students were only allowed to read a set speech to their subjects. They were not allowed to change a single word, and did not do so. Apparently, they managed to communicate their expectations through small changes in inflection of voice.

Reinterpretation Effect

One of the reasons study participants respond to observer expectation in medical studies is a desire to please their physician. Patients tend to stress improvements and downplay problems if that's what they sense the doctor wants to hear. This does not necessarily involve lying. Participants may simply reinterpret their own experience to show improvement. A good example of this reinterpretation effect occurs when you take vitamin C over the winter and then decide, no matter how many colds you had, you would have gotten more if you didn't use the vitamin C. You don't really know this, but you may tell yourself it is true nonetheless.

Placebo Effect

An entirely different possibility is that the power of suggestion may actually improve your health. This is the concept of the placebo effect. It may be, for example, that if you expect your knee arthritis to improve, it really will improve, through the power of the mind. (The concept of the placebo effect has recently undergone serious challenge, but it probably does occur at least to some extent.)

Memory Distortion

Memory distortion effects also influence the apparent outcome of treatments. Physicians (like everyone else) have a tendency to remember their greatest successes and most extreme failures, and drop from their memory everything in between. This can lead to a very skewed recall of the effectiveness of a treatment. Suppose a surgery works dramatically 15 times, fails absolutely 5 times, and yields mediocre results in the great majority of patients. The surgeon will most likely recall the surgery as highly effective.

Cognitive Dissonance

Cognitive dissonance is another influence that makes physician impression unreliable. It is a well-established principle of experimental psychology that if you state out loud that something is true (eg, a treatment is effective), your mind will jump through hoops to make you experience the results as consistent with your beliefs. If you make your living doing something, you will similarly experience a strong tendency to believe that what you are doing really works.

Natural Course of the Disease: Illusion of Agency

Another major confusing influence is the natural course of the disease. Many diseases eventually run their course and symptoms improve on their own. This can give a false impression that a treatment has worked. However, due to a very powerful psychological tendency called the illusion of agency, a doctor will tend to feel that her efforts caused this improvement.

Regression Toward the Mean

A related effect is called regression toward the mean. This term refers to a statistical principle. Simply put, most objective measurements of the state of the body fluctuate over time. Cholesterol level is a good example. People who are admitted to a study because their cholesterol levels are high may simply have high cholesterol at the moment they were tested for the study. During the subsequent several months, their cholesterol level will naturally move up and down. Suppose they happened to have been caught at a time of particularly high levels at the beginning of the study. By the end of that study, odds are they will show a lower reading. You might object that the effect should be symmetrical, and they just as well could have been caught at a low cholesterol moment at the beginning of the study. However, if that had been the case, they wouldn't have been allowed to participate in the study, because they wouldn't appear to have high cholesterol. Thus, this effect tends to produce an impression of improvement when in fact what is being observed is simply the workings of chance.

Study Effect

Another influence is called the study effect. Individuals in scientific studies (or under the care of a physician generally) often feel motivated to take better care of themselves overall. If you have diabetes, for example, and you enroll in a study of a new diabetes treatment, you may subconsciously begin to take your insulin shots more religiously, control your diet more enthusiastically, and make sure that you don't miss any doctors' appointments. The net result may be an improvement in symptoms that has nothing to do with a specific therapy under study.

Study Drop Outs

Finally, participants with bad results may drop out of a study (or stop coming to a physician), while those with good results remain. This will tend to bias the apparent outcome toward more positive results.

All of these factors combine to make it immensely tricky to informally discover whether a treatment is effective. Neither a physician's clinical experience nor a patient's personal experience is particularly trustworthy. By the 1960s, researchers had begun to settle on an effective solution to this problem.