I am a big fan of Consumer Reports magazine. For those of you who don’t know, it’s a consumer information and advocacy magazine. In each issue, it selects groups of products (e.g. computers, televisions, lawn mowers, refrigerators), and makes recommendations based on a battery of tests, surveys, and investigation of the product. It tests all of the products that it recommends, and usually has a very scientifically-minded approach to studying consumer goods.

However, the magazine is not limited to the analysis and recommendation of products, but also contains other supplementary material. Some of these things, like recall and safety notices, are often valuable in supplementing the content of the magazine. However, they can sometimes get into trouble in their supplementary sections, leading to problems like those in their latest issue. I took exception to two articles in particular: When to Buy Organic and How Did You Sleep Last Night?.

When to Buy Organic

On page 8 of this month’s issue of Consumer Reports is an article titled Fruits and Vegetables: When to Buy Organic. The article gives some useful information about organic food; mainly that it costs more. The purpose of the article is to list the fruits and vegetables that are more worthwhile to purchase organic.

On the whole, the article appears to be rather informative, but it seemed to try and avoid the most important question that must first be answered regarding organic food: Is it worth buying at all? I was surprised that never once did they question the underlying assumptions of the health benefits of organic food; those assumptions being:

1. That the pesticides used in growing non-organic foods are sufficiently harmful to the human body to cause adverse effects.
2. That the organic pesticides used in growing organic foods are less harmful to the human body than those pesticides used in growing non-organic foods.
3. That after being properly cleaned, a sufficient quantity of the pesticides remains on the fruit or vegetable, making it dangerous to eat.

At least it’s good to see that Consumer Reports went out of their way to get their data from the (obviously) non-biased “Organic Center, a nonprofit research and advocacy group”.

Unfortunately, as far as we have been able to determine, there are no adverse effects to humans from any pesticides left on fruits and vegetables after they have been washed. All evidence shows that the quantity and toxicity of any remaining pesticides on the products we eat is inconsequential to our health. Additionally, many consumers of organic food operate under the false assumption that organic foods are grown with no pesticides, when in fact the only difference is that the pesticides used in growing organic foods has been certified as organic. This doesn’t mean that the pesticides are any safer, despite what many people believe; at the end of the day, they are still compounds meant to kill any organic life that attempts to eat the plants. Fortunately for us, to kill a human, it takes significantly more poison than it does to kill an insect. There is even evidence that the more effective pesticides (and herbicides and fungicides) used in the production of non-organic crops are beneficial to our health, due to their ability to better kill harmful organisms and moulds.

To date, there is no conclusive evidence showing any harm from eating foods grown with pesticides. Furthermore, there is no evidence showing that eating foods grown with organic pesticides is any safer than eating food grown with non-organic pesticides. If there is no health benefit from eating organic food, then all you are doing is paying more for a food that isn’t necessarily healthier or of a higher quality¹, and taking away farmland from more productive farming techniques that help make food more affordable at home and abroad.

For a more detailed look at organic foods, and weighing the potential benefits against potential harms, there is an excellent article (written by an actual doctor, no less) over at Quackwatch.org.

How Did You Sleep Last Night?

Page 12 of the same edition of Consumer Reports contains an article titled How Did You Sleep Last Night?. This article is an interesting read, and offers some good insight into sleep problems, as well as some potential solutions. The methods investigated (prescription or non-prescription medication, sound machines, consistent sleep routine, relaxation) all seem like good methods; worthy of investigation. Overall, the article is quite informative and well done. I did however take issue with a survey that Consumer Reports published on page 15.

The survey is titled Techniques that Work Best.

To gauge the effectiveness of the most commonly used tactics ,we analyzed the experiences of 2,021 problem sleepers and confirmed insomniacs, who used a specific approach for at least eight days in the month before they were surveyed.

In principle, the surveys that Consumer Reports conducts are pretty well done, and this one is no exception. The problem I have with this survey is that the title is a bit misleading. The results of the survey do not actually tell us which techniques work best, but rather which techniques respondents thoughtworked best – an important distinction. Surveys are a great way of measuring consumer satisfaction with a product, but because they are prone to personal opinion and biases, they are not necessarily the best way to test the efficacy of a method that has objective results. The survey can only tell us which respondents thought they had a better night’s sleep due to the method they used, and not how effective the method was, or if it provided a better quality² of sleep.

The reason that the survey cannot tell us these things was that it was not a study, and as such, contained no placebo or control group, or objective measurement of the results.

A placebo-controlled trial is important for determining the efficacy of some type of intervention (medical or otherwise), because often the suggestion that a technique or medication will make somebody better is enough to make a person perceive an improvement in their condition. This can manifest as a reduction of subjective symptoms like pain, headaches, nausea, and sleeplessness (among other things), but this effect, called the placebo effect, has no objective health benefits. In other words, people might feel like they are sleeping better, but the duration and quality of their sleep might not have actually changed — and their health will still continue to suffer if their poor sleeping habits persist.

Control groups³ are also important parts of the testing process, because we want to try and isolate the change caused by the intervention from changes in the environment, or other any other possible causes for change. It is incorrect – and immensely unhelpful – to assume that if somebody achieves a positive result from a treatment, then the treatment was the cause of improvement. People are complex systems, and we live in complex environments. Without a control group, it is impossible to tell exactly what caused the improvement. It could have been the sleeping pills, but it also could have been a change in the respondent’s sleeping patterns or diet, or an environmental change like temperature or humidity. It also could have been a combination of factors. There are so many things that could contribute to this person’s improved sleep (assuming that it is an objective improvement, and not just a subjective one) that it would be impossible to isolate any one factor without a control group with which to compare the ‘positive’ result.

Overall, Consumer Reports is still a great magazine with a very scientifically-minded approach to consumer advocacy. These two problems are rather minor, and nowhere near the pseudoscience and mysticism promoted elsewhere, but they are still somewhat problematic. I would like to see Consumer Reports maintain its image as a respected, accurate, and critical magazine, and in order to do that, it has to put the same care into all of the information it publishes; not just the information on its featured products.