“Toxic Nation” or Junk Science?

One of the top items on CTV last night was an Environmental Defence report to be released today purporting to show that Canada is a “toxic nation” because “Canadians are walking around with a cocktail of harmful toxic chemicals in their bodies” (CTV News website). The story also appeared in The Globe and Mail.

But the “report” is based on blood and urine samples taken from a grand total of 11 people, including: Dr. Kapil Khatter, head of Canadian Physicians for the Environment; Merrell-Ann Phare, Legal Counsel and Executive Director of the Centre for Indigenous Environmental Resources; and David Masty, Chief of the Whapmagoostui First Nation

Wasn’t there a song that went, “a doctor, lawyer and an Indian chief couldn’t love you as much as I do?” Now we know why: it’s hard to get in the mood when PCBs and heavy metals are weighing down certain vital capillaries.

But seriously, the point here is that established media outlets should know better than to give such prominent play to a report that, besides coming from an environmental advocacy group, has little if any scientific value because it surveyed only 11 people. Such reports should get no more than a digest item in the print media. As for TV and radio newscasts that already have a hard time squeezing a lot of news into very little time, this report shouldn’t have even made the line-up.

Props to Andre Picard of the Globe for at least getting a comment from Health Canada:

Health Canada spokesperson Paul Glover said: “It’s only 11 people. It’s not statistically significant . . . but it is an indication and we will take a look at it.” . . . Mr. Glover said “obviously Canadians will be somewhat concerned. They didn’t choose to put chemicals in their bodies. So how did they get there? But for Health Canada the question is: What is the level of risk?”

With the rise of more and craftier advocacy groups, think tanks and PR strategies (and, if you believe it, the “death” of advertising), the media’s task of determining what is real and relevant is becoming harder at the same time that it is becoming more important. But surely they can do better than getting played by an advocacy group pushing a study of only 11 people?

What’s worse is that the media is not above playing these sketchy survey games themselves. In 1997, the Atkinson Foundation (i.e. The Toronto Star) funded “Speaking Out,” a study by the pink tank Caledon Institute. According to the paper setting out the study’s description, research strategy and methodology (available at http://www.caledoninst.org/ -- search publications for “speaking out”):

The Speaking Out Project was established in January 1997 to document the effects that policy changes are having on ordinary people in Ontario, especially those with low incomes. The project is a longitudinal study of 40 Ontario households, which will be interviewed intensively at six-month intervals until the year 2000. To supplement and interpret the households’ experiences, Speaking Out also analyzes a wide range of other sources of information. We will report our findings after each round of interviews.

The institute defended limiting the study to only 40 households (page 4):

We considered a number of different factors when deciding on our sample size. Too many households would have made proper tracking difficult and too costly; too few would have meant insufficient coverage of various characteristics. We decided that a sample of 40 households allowed adequate coverage of demographic and other characteristics, was manageable and accommodated the expected withdrawal of some households from the project.

Later on in the document, they reveal this little nugget (page 5):

While the smaller sample size means that we cannot automatically generalize our specific findings from these 40 households to all others in Ontario with similar demographic profiles, a range of complementary surveys and other types of studies and databases are available for our use. When the issues we explore overlap with other studies, that data will be investigated to see if generalization is possible. Thus, our analysis and reports will reflect a multimethod research approach.

By my reading, this means “our basic research is useless because our sample is too small, but we’re going to pad it out with other research we find if it confirms what our original useless research shows.”

No matter. In October 2000, The Toronto Star’s Caroline Mallan reported that the Speaking Out project had found that, despite Ontario’s economic boom, nearly half the households in the study group were worse off than they were in 1995, and another quarter of them were about the same. (I’m guessing that over a quarter of them were therefore doing better, but the Star’s story does not provide the remaining figures.)

The Star reported that the institute “melded its findings with Statistics Canada data to come up with a picture of low income people struggling to pay the bills” but since the referenced report is apparently not on the institute’s website, it’s not clear what data was “melded” or how. Could they have used StatsCan demographic data to generalize their findings, i.e. exactly what they said they couldn’t do on page 5 of their methodology?

Turning back to the toxic pretenders, I don’t know whether I have heavy metals or PCBs in my body, but I can say without benefit of a blood sample that the authors of Toxic Nation are overflowing with one unpleasant substance. I look forward to see whether “Toxic Nation” makes it into the Financial Post’s annual Junk Science Week next year. It certainly merits consideration.