Earlier this year, the media went crazy for a study that reportedly found that eating chocolate could accelerate weight loss. We’ve blogged time and time again about the media’s tendency to sensationalize scientific findings, overstating a study’s significance. In this case media outlets around the globe dropped the ball, failing to note significant flaws in the study, investigate the fact that the study’s main author and organization didn’t really exist, or consult any outside scientists to evaluate the study’s strengths and weaknesses. And it turns out, the entire study was conducted as a hoax to uncover these very flaws in science journalism.

The study’s author, Dr. John Bohannon, revealed this week how he and a few colleagues conducted the study to highlight both the flaws in nutritional science and problems with how science journalists report on new research.

Dr. Bohannon’s article highlights why his study had severe flaws:

• He only used 15 research subjects. (Read why sample size matters here.) He divided these subjects into 3 groups: a control group instructed to eat their normal diets, a group instructed to eat a low-carb diet, and a group that ate a low-carb diet plus chocolate.
• He tested 18 different markers in these test subjects—weight, cholesterol, sleep quality, etc.—but only reported on the findings that were “significant” for the chocolate group. As he explained, “We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.”
• The weight fluctuations between the control group who ate their normal diet, the low-carb group, and the low-carb plus chocolate group were minor—roughly 5 pounds over the 21 day study. To put that in perspective, Dr. Bohannon points out that the average woman’s weight fluctuates about that much during their menstrual cycle.

Despite all these flaws, his article was readily accepted within 24 hours by several scientific journals. (Read more about these pay-for-play journals here.) Once it was published, he worked with a PR specialist to craft a press release designed to catch reporters’ eyes. Among the alarming observations he made about the journalists who interviewed him about the study:

• No one asked how many test subjects they used;
• No one contacted outside researchers to review the study;
• No one wondered whether it had undergone peer review;
• No one quested the fact that Dr. Bohannon’s “Institute of Diet and Health” was a recently created shell website.

Dr. Bohannon’s hoax highlights why it’s so hard to trust stories you read about “new research” touting a food’s health benefits or possible health problems. Nutritional science is far from exact, and even the most rigorous and well-designed studies are hard to trust. This is a pressing problem, considering the federal government is in the midst of updating its “dietary guidelines” based on largely unreliable nutritional research.

We hope Dr. Bohannon’s experiment will encourage reporters to dig deeper when they receive a press release about a new study. In the meantime, we’ll keep eating chocolate just in case it helps us lose weight.