When technology undermines science

On the day the iPad was launched, Apple sold over 300,000 of the tablet computers. Since then, over 3 million iPads have been sold. Our society is infatuated with technology, and this affects us in ways both obvious and subtle. Here I want to examine how our adoration of technology influences the way we think about science, and in turn how we see our whole world. I should note that I write this as a scientist, and someone with a long-time interest in, and fascination with technology.

Because the words science and technology are often paired, their meanings tend to be conflated. But science can be pursued with few technological spin-offs (as is the case with astrophysics, for example) and technology can be developed without the use of science (as was the case with the early technologies developed by trial and error in prehistory). Certainly scientific discoveries can often be used to develop new technologies, and existing technologies can be evaluated scientifically. But science itself is not about developing technology, it's about learning through systematic observation and (sometimes) experimental manipulation. Some would argue that this distinction is mere semantics, but I will argue that confusion between science and technology leads to some very unfortunate consequences.

Because we're so taken with technology, and because of the close connection between science and technology, it's not surprising that science is held in high esteem. But this is a double-edged sword. The downsides of technology (e.g. sedentary behaviour patterns and burgeoning rates of obesity, global warming from carbon emissions, toxic waste, etc.) are sometimes blamed on science. On the one hand, this is fitting: for better or worse, without science, modern technologies could never have been developed. On the other hand, surely it is our society's moral, economic, and political choices that determine how scientific knowledge is applied, and responsibility for those choices should fall to the decision makers. But our terminological confusion blurs such distinctions, and science and technology are routinely seen as one and the same. Praise or criticism of one is seen as identical to praise or criticism of the other. This has lead to a curious polarization of views.

The church of science

On the one hand, a triumphalism of science has become more and more common. Science is increasingly seen as providing the most trustworthy information, or perhaps the only reliable source of knowledge, not just about the physical world, but about all aspects of life.

I believe two factors underlie this tendency. First, the products of technological wizardry provide a concrete demonstration of the mastery and control that scientific knowledge can provide. The most important point here is the universality of this demonstration: no special knowledge or education is required to appreciate the power of technology. This technological factor rides on top of an epistemological claim. As Luke Muehlhauser puts it:

the massive success of science leads me to suspect that methods condoned [sic] by science are the most successful methods of knowing we have discovered yet.

And while it seems likely that a philosophical argument such as this will only appeal to a limited audience, it nevertheless provides the intellectual muscle beneath the alluring skin of technology.

Triumphalism about science has a long historical lineage, expressed in the first half of the 20th century in the school of logical positivism, and more recently in some of the writings of the so-called new atheists. In their extreme forms, such arguments tend towards scientism, the view that only scientific statements have any meaning and that, ultimately, science will provide all the answers. The trouble is, if science is seen as having all the answers, it must either expand to encompass a much broader range of concerns, or else dismiss such concerns as meaningless. Where does that leave ethics, philosophy, literature, history, art? While science can inform each of these fields, a radical redefinition of science would be required to assimilate them. And yet that is just what is being proposed.

Philosophy. Luke Muehlhauser argues: "I think philosophy will be most productive when it functions as an extension of successful science ... ". Commenting on such thinking, Massimo Pigliucci writes:

There are profound differences in method, style and type of problems between science and philosophy, and frankly I think that people who deny or minimize this simply have not taken their time to read any philosophy, or they would immediately see how bizarre it is to deny the difference.

More broadly, I am having a really hard time understanding the agenda of people here who wish at all costs to dismiss philosophy or absorb it into science. Why are you so bent on arrogating more epistemological power to science than it possesses? Why is it not good enough to say that science is by far the best approach we have devised to understand the natural world, but that there are problems that lie outside of it and other disciplines that are better equipped to address those problems?

Ethics. Sam Harris recently gave a TED talk titled “Science can answer moral questions”, in which he argues that "The separation between science and human values is an illusion". Massimo Pigliucci described the "malady that strikes Harris: scientism, the idea that science can do everything and provides us with all the answers that are worth having." Thinkmonkey, a commenter on Pigliucci's blog wrote:

Sam Harris has simply not done the hard work needed to understand the historical and ongoing arguments in ethical theory and metaethics - the context in which the argument he wishes to make *must* be situated. Perhaps these arguments have not settled very much, but they have at least established some shared terminology and made important distinctions: Without knowing the terminology and understanding the important distinctions (and the reasons for them), Harris cannot help but be confused - and to introduce still more confusion when he attempts to engage with his critics.

Philosophy may be where all the unanswered questions live, and may not get a lot of respect thereby, but at least we try to avoid these kinds of messes. Or, as Sydney Morgenbesser famously described our collective work: "You make a few distinctions. You clarify a few concepts. It’s a living."

The Humanities. The academic disciplines concerned with the human condition include history, literature, law, languages, art, and religious studies. Aspects of these and related fields may be studied using the methods of social science. But large parts of these disciplines use methods that are not scientific. Criticism of these disciplines is increasingly common. For example, the website of Edge: The Third Culture sneers:

The third culture consists of those scientists and other thinkers in the empirical world who, through their work and expository writing, are taking the place of the traditional intellectual in rendering visible the deeper meanings of our lives, redefining who and what we are.

... the traditional American intellectuals are, in a sense, increasingly reactionary, and quite often proudly (and perversely) ignorant of many of the truly significant intellectual accomplishments of our time. Their culture, which dismisses science, is often nonempirical. It uses its own jargon and washes its own laundry. It is chiefly characterized by comment on comments, the swelling spiral of commentary eventually reaching the point where the real world gets lost.

Mathematics. Interestingly, the claim that science provides the only reliable source of knowledge is easily refuted. Mathematics uses deduction to arrive at certain knowledge, something that science cannot achieve. One response to this is to claim that mathematics is part of science. Certainly mathematics is a key tool of science, but claiming that it is part of science goes too far. A different response is to point out that mathematical knowledge pertains to abstract entities, and thus in itself is not practical. This is indeed correct, but it highlights the key point that there are different kinds of knowledge, which cannot be seen as competing, because they belong to entirely different spheres.

Science is sometimes identified tout court with rationalism, and in a you're-either-with-us-or-against-us manoeuvre, everything else is simply deemed to be irrationalism. This is more a rhetorical trick than a line of reasoning, but once again we see the definition of science being expanded at will.

Science as fiction

At the other extreme, is an anti-science sentiment that manifests itself in support for pseudoscience, quackery, and superstition. From Deepak Chopra to crystals to the anti-vaccination movement, anti-science thinking is surprisingly prevalent. As I suggested previously, some of this is a reaction against the evident problems engendered by technology, coupled with a confusion between science and technology. But some of the anti-science thinking is a reaction to the kind of triumphalism of science that I have described.

What is to be done?

I've argued that fuzzy definitions have done real damage, fueling a grandiose vision of science and its flip side, a crude resurgence of superstition and anti-science thinking. The pairing of science and technology is here to stay, and the allure of technology will continue to promote an exaggerated conception of science. What can be done in the face of this tendency?

First, it remains important to distinguish between science and technology. The careless fusing of the two terms contributes to the unwarranted expansion of the notion of science. Second, it is important to challenge attempts to expand the purview of science to non-empirical matters such as ethics. This does no good to either science or ethics. While science can certainly inform ethics, the primary focus of ethics is normative, not predictive or explanatory. Science provides the best way to understand the physical world, but it is not a source of values or meaning. Third, pseudo-science, superstition, and quackery should be challenged by insisting on high-quality evidence. However it should be remembered that such delusions are nourished by out-sized claims about the universal dominion of science. Attempts to discredit new-age nonsense can backfire when metaphysical claims are denounced as being unscientific. Science can only address empirical claims. Non-empirical claims may certainly be challenged, but not by science.

Mind your own business

Many of the issues I have discussed are particularly vexing where the mind is concerned. Advances in neuroscience have encouraged a physicalist view that in its most extreme form argues that the mind is nothing more than the activity of neurons. This idea has an interesting connection with technology. Early computers were described as being "like a brain". As computers became more familiar, the simile was inverted, and the brain was seen as being "like a computer". More recently this process has reached its conclusion, and it is common to hear that the brain simply "is a computer".

Of course it's true that the brain computes, albeit in a way rather different from our digital computers. But somehow, along with the computation, we experience consciousness, a sense of self, the impression of free will. We experience sensation (rather than simply processing signals), we feel emotion, we delight in beauty and we abhor ugliness. Questions about these aspects of mind have occupied philosophers from the earliest times. Naturally, developments in the scientific understanding of the brain have had an important impact on philosophy of mind. But the fundamental questions remain.

Unfortunately, reductionist views about the mind are flourishing, nourished by both enthusiasm about developments in neuroscience and uncritical acceptance of the technological metaphor that "the brain is a computer". It is perhaps noteworthy that Sam Harris, who argues in his new book that Science Can Determine Human Values, has a PhD in neuroscience. In a New York Times review of Harris's book (with the telling title Science Knows Best), Kwame Anthony Appiah writes:

when he stays closest to neuroscience, he says much that is interesting and important ... Yet such science is best appreciated with a sense of what we can and cannot expect from it ...

Indeed we should approach all science this way.

Why do we overinterpret study findings?

MSNBC recently reported that a new study suggests "U.S. states whose residents have more conservative religious beliefs on average tend to have higher rates of teenagers giving birth". (I learned of this on Rationally Speaking.) The study itself is Open Access, so all the details are freely available. The scatterplot illustrates the strong association the authors found. Now, the authors were reasonably cautious in how they interpreted their findings. The trouble is, the general public may not be.

A common error is to conclude the study shows that religiosity causes higher teen birth rates. But correlation does not imply causation. It could be that higher teen birth rates cause religiosity. Or perhaps a third, unidentified factor causes both.

But isn't the strength of association still impressive? It is. But what if, as I just suggested, there are other variables involved? Such confounding variables (or confounders, as they are commonly known) can wreak havoc on this sort of analysis. Indeed, the authors of the study did adjust for median household income and abortion rate (both at the state level). But it is possible that other confounders are lurking. And unfortunately, we tend to forget entirely about the possibility of confounders when we hear about study findings.

Another error is to conclude that the findings directly apply to individuals. Here I will quote the authors directly:

We would like to emphasize that we are not attempting to use associations between teen birth rate and religiosity, using data aggregated at the state level, to make inferences at the individual level. It would be a statistical and logical error to infer from our results, “Religious teens get pregnant more often.” Such an inference would be an example of the ecological fallacy ... The associations we report could still be obtained if, hypothetically, religiosity in communities had an effect of discouraging contraceptive use in the whole community, including the nonreligious teens there, and only the nonreligious teens became pregnant. Or, to create a different imaginary scenario, the results could be obtained if religious parents discouraged contraceptive use in their children, but only nonreligious offspring of such religious parents got pregnant. We create these scenarios simply to illustrate that our ecological correlations do not permit statements about individuals.

To err is human ...

My goal here has not been to criticize the authors of this study, nor the media. Rather, what I find remarkable is how such a simple statement—"states whose residents have more conservative religious beliefs on average tend to have higher rates of teenagers giving birth"—can be so easily misinterpreted, and in so many different ways! Does anyone know of any research about our tendency to overinterpret scientific findings? Of course, we'd probably overinterpet it.

Absence of evidence ...

In a valid deductive argument, the conclusions follow necessarily from the premises. This is a proof in the mathematical sense of the word. Provided we know that the premises are true, we can establish with complete certainty that the conclusions are true. For example, identifying a single unicorn would establish without a doubt that unicorns exist.

Unfortunately much of the time this type of certainty isn't possible. Consider another example from the realm of mythology: weapons of mass destruction (WMD) in Iraq. Here's what Donald Rumsfeld had to say on the subject in 2002 (the boldface is my addition):

There's another way to phrase that and that is that the absence of evidence is not evidence of absence. ... Simply because you do not have evidence that something exists does not mean that you have evidence that it doesn't exist.

But surely hunting high and low for WMD month after month and not finding any (absence of evidence) supports the inference that there aren't any there (evidence of absence). Indeed, it turns out that the popular maxim cited by Rumsfeld is simply incorrect.

But didn't he have a point? Absolutely: failing to prove that something exists does not prove that it does not exist. Or, in the words of the English writer William Cowper (1731-1800):

Absence of proof is not proof of absence

Compare this with the version invoked by Rumsfeld:

Absence of evidence is not evidence of absence

The originator of this maxim seems to be the cosmologist Martin Rees, although it has been attributed to many others, including Carl Sagan. By substituting the word evidence for proof it makes a much stronger (and invalid) claim. Evidence, after all, is often uncertain. If I look outside and see that the ground is wet, that is evidence that it has been raining. But perhaps my neighbour was watering her flowers. Seeing someone walk by with an umbrella folded under their arm might strengthen my evidence for the rain hypothesis, but perhaps they are anticipating rain later on. In general, evidence can support an inference, but it won't necessarily prove it. And that's where Rees's formulation of the maxim falls down.

Black and white thinking about evidence

When evidence is construed as being certainty, we get into all kinds of trouble. This is how Rumsfeld turned a simple truism (no WMDs have been found, but they might still be) into a puzzle of obfuscation (absence of evidence is not evidence of absence).

But Rumsfeld is not the only one. As I noted recently, the term "no evidence" is commonly used to describe situations where an effect is not found to be statistically significant. Now statisticians are wary of people concluding that a lack of statistical significance implies that there is "no effect". (It might be, for example, that the sample size was inadequate.) Hence, it is not at all uncommon for statisticians to declare that absence of evidence is not evidence of absence! As Kim Øyhus has pointed out, even the American Statistical Association buys into it, as the t-shirt they sell attests.

Of course statisticians know well that uncertainty isn't easy to think about or communicate to others. So why have we fallen into this trap?

Well, part of the reason may be philosophical. Statistical reasoning is inescapably inductive—it does not guarantee certainty. Philosophers have been worrying about what is called the problem of induction for a very long time. David Hume (1711-1776) challenged the logical foundations of induction, and ever since, philosophers have sought a way around the problem. The reigning "solution" is known as the hypothetico-deductive method, developed by philosopher of science Karl Popper (1902-1994). Popper argued that induction in science could be avoided by proposing a hypothesis and then seeking evidence that would either prove the hypothesis wrong ("falsify" it) or fail to do so. This is very similar to the frequentist statistical hypothesis testing framework that developed from the work of Fisher, Neyman, and Pearson. Unfortunately, it lends itself to black-and-white thinking. A hypothesis is either proven wrong or it isn't. There's no grey zone.

Popper's formulation, in particular, buries the uncertainty completely, construing the reasoning as entirely deductive. Suppose, for example, that a new biochemical theory predicts that a certain drug will shorten the duration of an illness, whereas the older theory does not. Now duration of illness depends on numerous factors, including differences in patients' immune systems, and we expect to see variation above and beyond any differences due to the drug. A clinical trial may demonstrate that the average duration of illness for patients who are randomly assigned the drug is shorter than that for patients who receive placebo, and that this difference is statistically significant at the 0.05 level. Has the older theory been proven incorrect? Not with absolute certainty. The evidence against it may be strong but it is possible that this is a "type-I" error—rejecting the null hypothesis even though it is true. Indeed, because of the way the statistical test has been designed, when the null hypothesis is true we expect to see such errors 5% of the time. The companion to the type-I error is the type-II error—failing to reject the null hypothesis even though it is false.

Pretending that type-I and -II errors don't exist is wishful thinking. Just as diagnostic tests produce false positives and false negatives, statistical hypothesis tests can give the wrong answer. The point is that we can study and control the error rates and make inferences while acknowledging their limitations.

It suited Donald Rumsfeld's purposes to be fuzzy about the distinction between evidence and proof. It doesn't suit ours.

Update 03-Jun-2009: I had originally attributed the maxim "Absence of evidence is not evidence of absence" to Carl Sagan in his 1995 book The Demon-Haunted World." Apparently however, the originator was cosmologist Martin Rees. There is reference to it in the proceedings of a 1972 symposium titled Life Beyond Earth & The Mind of Man [pdf], jointly sponsored by Boston University and NASA. In his introductory remarks, the chair, Richard Berendzen stated:

A generation ago almost all scientists would have argued, often "ex cathedra," that there probably is no other life in the universe beside what we know here on Earth. But as Martin Rees, the cosmologist, has succinctly put it, "absence of evidence is not evidence of absence." Beyond that, in the last decade or so the evidence, albeit circumstantial, has become large indeed, so large, in fact, that today many scientists, probably the majority, are convinced that extraterrestrial life surely must exist and possibly in enormous abdundance.

(The boldface is mine.) Note that Carl Sagan was one of the panelists at the symposium.

Something fishy about "no evidence"

Searching Google for "no evidence" yields "about 21,300,000" hits. It seems we're keen to deny that there is any empirical support for countless different claims. For example, scientificblogging.com reports that there is "No Evidence For Fish Oil Benefit In Arrhythmias" based on a systematic review just published in BMJ. (Full disclosure: I have previously participated in research on omega-3 fatty acids, however I have no related financial interests.) What does the review itself say?

This is the first systematic review attempting to evaluate whether the protective mechanism of fish oil supplementation is related to a reduction of arrhythmic episodes determined either by a reduction in implantable cardiac defibrillator interventions or a reduction in sudden cardiac death. We found a neutral effect on these two outcomes. The confidence intervals for these outcomes were wide and a beneficial effect up to a 45-48% relative risk reduction cannot be excluded.

To better appreciate this, here's their Figure 2:
Note that of the three studies that looked at the proportion of implanted defibrillators that were triggered, one showed a statistically significant effect in favour of fish oil, and the other two did not show statistically significant effects (one favoured placebo and the other favoured fish oil). Six studies looked at sudden cardiac death in patients taking fish oil compared to those taking placebo. Only one was statistically significant, and it favoured fish oil. Of the five studies that did not show statistically significant effects, two favoured fish oil, and three favoured placebo.

The diamond shapes in the figure show the pooled estimates with their 95% confidence intervals: in each case the diamond overlaps an odds ratio of 1, indicating that the overall effect is not statistically significant. And when there's a non-statistically significant effect, it is common practice to say there is "no evidence". But that can be very misleading! After all, two of the individual studies did show a significant benefit of fish oil. So what's going on? Well, for starters, there's some indication of heterogeneity between the studies (particularly in the case of the defibrillator studies). But it also seems that more large studies are needed: a good deal of the variation in results between the studies may simply be due to the play of chance. Quite substantial benefits of fish oil are entirely plausible: relative risk reductions of as much as 45-48%!

What is "no evidence"?

Consider the figure below:

At the bottom there is a gray axis line with tick marks and a vertical gray line indicating the "null" value (where there is no preference one way or the other). The blue line with arrows at each end represents an infinitely wide confidence interval. This is the most straightforward representation of "no evidence": there is simply no empirical information to indicate what the true effect might be.

But suppose we have a very small sample, that is, one that provides almost no empirical evidence. The figure might become:

The only difference is the blue dot on the confidence interval just a bit to the right of the null line. It represents the point estimate based on a very small amount of empirical information. Of course it could equally well have been on the left hand side (or perhaps directly on the null line). Regardless, the confidence interval is still very wide, so very little can be said about the true effect. With such a wide confidence interval, the location of the point estimate is almost irrelevant.

Finally, suppose that a reasonably large sample is available:

I have left the point estimate at the same place. The confidence interval no longer has arrows on either end and is relatively narrow. However it still overlaps the null line. That means the estimate is not statistically significant. Sometimes this sort of situation is described as showing "no evidence of an effect". But as I noted above, that's quite misleading language. In fact, what this situation shows is indeed evidence—evidence that any effect likely has a magnitude of no more than two tick marks (whatever they represent) on the right hand side of the null line or a magnitude of no more than about a half a tick mark on the left hand side of the null line.

But here's the tricky part: what do those ticks represent? Suppose the axis represents annual cost savings that might result from implementing a certain type of federal government program. If each tick mark represents $1000, then we have estimated that the program will cost at most $500 a year and save at most $2000 a year. In other words, the program has been shown to be effectively revenue neutral: the evidence suggests that the cost/cost-savings of the program will not be important. On the other hand, if each tick mark represents one million dollars, most of us would feel that the jury's just not in yet. A possible cost of $500 a year is a drop in the bucket, but $500,000 a year is something else entirely.

In a way, money is the easiest measure to evaluate like this. Things like safety are much harder. For example, if the evidence suggests that a certain chemical may increase the rate of certain types of cancer, but the findings are not statistically significant, what can we conclude? Can the manufacturer claim that there's "no evidence" the chemical is harmful? Can health activists claim that there's "no evidence" the chemical is safe?

I would argue that the term "no evidence" is inappropriate in either case. The underlying questions remain: what is required in order to conclude that a chemical is harmful or that it is safe? Ultimately there's no getting around the issue of how large a difference (in, for example, cancer rates) has to be in order to be considered important. And that's a rather uncomfortable question.

Stop worrying and learn to love the chemicals

Well Margaret Wente is at it again. In a column last week titled Yellow duckies and other killers, she claims that "Mothers across Canada have been prostrated by the plastics scare."

It's hard to be a good mother these days. Deadly perils lurk everywhere. Take that yellow bathtub ducky, contaminated with a dangerous substance known as BPA.

But why stop there? Wente proceeds to list other putative hazards: toxic mould, pesticides, perfumes, "death-rays from the sun", walking barefoot in the grass. The message is clear: stop worrying already!

We forget how negligent our own parents were. They gave us naked sunbaths and let us suck on plastic duckies and roll around on pesticide-drenched lawns. It's astonishing how ignorant they were, and how many of us managed to grow up.

Now Margaret Wente is no scientist (what was your first clue?), so she needs an outside authority:

Dr. Elizabeth Whelan is president of the American Council on Science and Health [ACSH], an independent group devoted to accuracy in health reporting. She points out that both BPA and phthalates have been studied intensively for decades. There are no studies - none - that show any link between these substances and harm to people. The basis for the claims of danger are all from studies done on rats, and they don't predict human risk.

According to Media Transparency, ACSH haven't disclosed their corporate donors since the early 1990's, but their 1991 annual report listed each of the following as contributing at least $15,000:

American Cyanamid Company * Anheuser-Busch Foundation * General Electric Foundation * Rollin M. Gerstacker Foundation * ICI Agricultural Products, Inc. * ISK Biotech Corporation * Kraft, Inc. * Monsanto Fund * The NutraSweet Company * John M. Olin Foundation, Inc. * Pfizer, Inc. * Sarah Scaife Foundation Incorporated * The Starr Foundation * Archer Daniels Midland Company * Carnation Company * Ciba-Geigy Corporation * Ethyl Corporation * Exxon Corporation * General Mills, Inc. * Heublein Inc. * Hiram Walker-Allied Vintners * Johnson & Johnson * Kellogg Company * The Esther A. and Joseph Klingenstein Fund, Inc. * Malaysian Palm Oil Promotion Council * National Starch and Chemical Foundation, Inc. * PepsiCo Foundation Inc. * Union Carbide Corporation

The under-$15,000 list continues on, listing all kinds of industrial, pharmaceutical, and food corporations.

Figures don't lie ...

In her April 19th column, Wente quoted an organizations called the Statistical Assessment Service (STATS) who similarly dismiss concerns about BPA. While they don't accept industry money, STATS is funded by a number of the same conservative organizations as ACSH. I think I see a pattern here.

But so what if these organizations get "conservative" funding? An anonymous commenter on my previous post wrote:

Why should the funding source matter? Isn't it the quality of the evidence and the arguments made? Your smear is the equivalent of an ad hominem attack.

My response:

I don't think it's ad hominem. If a medical study was funded by a pharmaceutical company, I'd like to know that. Not that it invalidates the study: as you say, the quality of the evidence and the arguments (analyses) made is centrally important.

So let's have a closer look at the quality of the evidence and arguments in one particular case.

I looked at a recent post on the STATS blog concerning formula- versus breast-feeding. While the author allows that "Yes, there is robust evidence that nursing reduces ear infections [otitis media] and diarrhea", he sets out to discredit claims of a link between formula feeding and diabetes, leukemia, and serious respiratory infections. In the latter case, he writes "The most recent research does not support the contention that formula carries a higher risk," citing a 1995 paper from the Journal of Pediatrics.

Interestingly enough, that study was supported in part by the Mead-Johnson Nutritional Group. Leaving that aside, however, here are some results from the abstract:

In the first year of life the incidence of diarrheal illness among BF [breast fed] infants was half that of FF [formula fed] infants; the percentage with any otitis media was 19% lower and with prolonged episodes (>10 days) was 80% lower in BF compared with FF infants. There were no significant differences in rates of respiratory illness; nearly all cases were mild upper respiratory infections. ... These results indicate that the reduction in morbidity associated with breast-feeding is of sufficient magnitude to be of public health significance.

Sure enough, they didn't find statistically significant differences in rates of respiratory illness. Now an important consideration in statistics is the power to detect differences, which is determined by a number of factors including sample size. So what was the sample size in this study?

... morbidity data were collected by weekly monitoring during the first 2 years of life from matched cohorts of infants who were either breast fed (N = 46) or formula fed (N = 41) until at least 12 months of age.

So there were a total of 87 infants. In their discussion, the authors write:

We did not observe any significant differences in the incidence or prevalence of respiratory illnesses between BF and FF infants. However, the vast majority of episodes were mild upper respiratory illnesses. Previous studies have indicated that the protective effect of breast-feeding is greatest for lower respiratory illnesses. The sample size in our study was not large enough to detect differences in more severe respiratory illnesses.

Blind trust?

Ultimately, we all have to rely on some surrogate measures to judge the quality and trustworthiness of the information we encounter. Our own expertise can only be so broad and we rely on others to help us interpret the world. Oldly enough the words of Ronald Reagan come to mind: "Trust, but verify."

A non-profit, non-partisan organization

In the April 19th edition of the Globe and Mail ("Canada's National Newspaper") columnist Maragaret Wente had a piece titled "The great plastics panic".

Wente reports that at an elementary school near where she lives, plastic water bottles have been "banished":

The kids know what's at stake. Plastic is death! At home, their anxious parents have stopped microwaving with plastic wrap. They've thrown out their plastic baby bottles and replaced them with ones made of glass. Leading retailers ... have banished plastic containers, baby bottles, sippy cups and pacifiers containing one offending chemical from the shelves. No wonder. A barrage of media reports have warned that the chemical in question - bisphenol A, or BPA - may be linked to breast and uterine cancer as well as lowered sperm count, early-onset puberty, obesity, hyperactivity, miscarriages, diabetes and other horrors.

"So," she asks, "how worried should you be?"

"On my list of a thousand things to worry about, BPA would rank about 892nd," says Trevor Butterworth, who's with an independent outfit called STATS (for Statistical Assessment Service). STATS is a non-profit, non-partisan U.S. group that analyzes the use and abuse of science and statistics in the media.

Although Butterworth's 892/1000 is obviously a rhetorical device, it still gets across the message there's nothing much to worry about. But there's another message: this is a quantitative guy! He works for an organization called the Statistical Assessment Service and accordingly he slings around numbers like nobody's business.

When I read this, I headed to the internet to check out this organization. Well, they have a pretty slick website. They describe themselves like this:

Since its founding in 1994, the non-profit, non-partisan Statistical Assessment Service (STATS) has become a much-valued resource on the use and abuse of science and statistics in the media. Our goals are to correct scientific misinformation in the media resulting from bad science, politics, or a simple lack of information or knowledge; and to act as a resource for journalists and policy makers on major scientific issues and controversies.

As a mark of our success, STATS' work has been featured on NBC's "Nightly News," "The NewsHour with Jim Lehrer" and ABC's "20/20" - and in print by The New York Times, Wall Street Journal, Washington Post, US News and World Report, New Scientist, New England Journal of Medicine, and many other publications.

Furthermore, "In 2004, we became an affiliate of George Mason University in Virginia." Pretty impressive.

But how is the organization funded?

STATS is a non-profit, nonpartisan organization that relies on philanthropic donations to support its operations. We do not take money from industry or industry-related groups.

And here, I must admit, I stopped. After all, I had read that STATS is:

• non-profit and non-partisan
  
• "independent"
  
• affiliated with a public university
  
• not funded by industry or industry-related groups
  
• "a much-valued resource on the use and abuse of science and statistics in the media"
  
• getting their work into the New England Journal of Medicine and New Scientist
  

I proceeded to read the rest of Margaret Wente's article. Butterworth's viewpoint was presented a number of times:

Mr. Butterworth maintains that most of the media have been reporting only one side of the story - the side that's driven by a handful of activist scientists and advocacy groups, such as Environmental Defence. Independent assessments conducted by food safety authorities in Europe and Japan, as well as various other risk assessments, have found no basis for the BPA scare. "We've had five major academic independent evaluations of the BPA risk over last two or three years, and they all keep saying the same thing," says Mr. Butterworth. "But they never get reported."

And what about the evidence from animal studies?

"The biological pathways in rats and people are different," notes Mr. Butterworth.

And the final word goes to ... Mr. Butterworth:

"Letting your child outside the door to breathe in exhaust fumes is more risky than letting them drink from plastic bottles," says Mr. Butterworth. He suggests if you're really worried about plastic, give up plastic bags. They suffocate 25 children a year.

Hmmm ... so maybe BPA is not so bad after all.

Uh, not quite ...

Just the other day I got some new insight into the Statistical Assessment Service, thanks to Wikipedia. (I wonder why I missed this the first time round.) Their entry about STATS includes a section on funding:

While the STATS website does not describe its funding sources, STATS is funded by a variety of conservative organizations, including Richard Mellon Scaife's Carthage Foundation, the Sarah Scaife Foundation, the Earhart Foundation, John M. Olin Foundation and the Castle Rock Foundation.

In the United States, funding information is available from the tax returns of 501(c)(3) nonprofit organisations. The information above was collated by Media Transparency. Another organization that looks into media manipulation is SourceWatch, who write:

STATS is a 501(c)3 non-profit organisation but its 2006 annual return to the Internal Revenue Service states that "salary costs for the organization are shared with the Center for Media and Public Affairs. CMPA ... reports the salary costs and files payroll reports under its tax identification number. DCFC is a related organization."[1] (It is not clear what "DCFC" refers to). The report also states that the relationship between STATS and CMPA is one of "common control".[2] Since STATS shares the offices (in the pricey "K Street" lobbying district of Washington) and staff of CMPA, it should be considered as a front, rather than a subsidiary or spin-off.

The Center for Media and Public Affairs is a topic in its own right (see here, here, and here). In 2001 some of the people involved in STATS and CMPA published a book called It Ain't Necessarily So. This review from Salon.com concludes:

A fair review of the state of science journalism is always welcome, but this cleverly disguised example of corporate propaganda isn't it.

Up Front?

The staff list of the Statistical Assessment Service is interesting. The president is S. Robert Lichter (one of the authors of "It Ain't Necessarily So"), who has been the DeWitt Wallace Chair in Mass Communications at the American Enterprise Institute and paid consultant to Fox News. There is a PhD economist and a PhD mathematician. These presumably constitute the core of the analytical team. The executive director is an MBA. And there are two journalists, one of whom is Trevor Butterworth.

Butterworth has no scientific training to speak of (rather, his training is in philosophy and intellectual history). Yet he makes quite strong statements about BPA. Consider this paragraph from Wente's column:

Does this mean BPA is completely off the hook? No. Lots of people think it needs more study. "The possibility that human development may be altered by bisphenol A at current exposure levels cannot be dismissed," said an important U.S. toxicology report this week. Some media stories billed this statement as a five-alarm fire. But as Mr. Butterworth says: "It's a very mild caution. Essentially, it says there is possibility there may be some effects, but we need more research."

Presumably someone with scientific training came to this conclusion and Butterworth is simply repeating it. But if Butterworth is simply a talking head, why is Wente not going to the source?

Toxic?

I started out wondering about the toxicity of BPA, and I still am. But along the way, I bumped into a different toxin altogether. So let's see what an Angry Toxicologist has to say about this (and see Butterworth's extensive comments in response).

Update: It turns out that in 2002, the Fraser Institute (a conservative think tank based in Canada) launched CANSTATS, clearly using STATS as a model. (By the way, the Canadian government's official statistics agency, Statistics Canada, is commonly referred to as StatsCan.) It seems that CANSTATS is no longer operating, but while it did it employed some familiar tactics.

It's complicated ...

It has been noted that when trying to explain almost anything, I have a habit of declaring—"It's complicated ..." And of course, it is (whatever it is). But is that just a cop-out?

Complexity is particularly challenging when decisions have to be made. There may be many factors beyond our control, that we understand poorly, or that we're not even aware of. And how these factors interact is often unclear. Complexity brings with it uncertainty, and uncertainty is always unsettling. One solution is to do nothing, and sometimes that's the best choice, as expressed in the aphorism "first do no harm". But, as in the case of climate change, the decision to do nothing ("more studies are needed") is often a poor choice.

Coping with complexity

Faced with complexity, we simplify. That is, we employ models that make it easier to think about the situation. This comes so naturally to us that we're often completely unaware that's what we're doing. For example, when we are "behind on sleep", we need to "catch up". This metaphor could be called SLEEP IS A RACE. The complexities of human sleep requirements are reduced to simple addition and subtraction, represented in terms of a racetrack (essentially a number line). This is an example of a mental model.

Mental models are more slippery than scientific models because they tend to be much less explicit. Scientific models are published and debated and put to empirical test. But we are embedded in our mental models, and like embedded journalists, our objectivity is profoundly compromised. And yet we can't do without mental models, any more than we can do without scientific models.

A double-edged razor

A well-known principle in science is Occam's razor, which argues for simplicity in modeling. But simplicity can go too far. Albert Einstein's take on this was:

Everything should be made as simple as possible, but no simpler.

An extreme case of simplification is polarized thinking. George W. Bush's "You're either with us or against us" is an apt example. Thinking like this both justifies and perpetuates violent conflict. Bush's statement is an example of what I might call a polarized moral or perhaps theological model.



Logic's limits

Even when we're aware of our models, it's easy to mistake them for reality. This applies not just to the elements of a model, but also to deductive inferences (which I'll refer to simply as logic) obtained in the context of the model. Logic has a seductive appeal: it offers certainty provided we observe some elementary rules, known since at least the time of Aristotle. There's just one hitch: our model has to be correct.

In fairly trivial situations, it may be possible to know our model is correct. One example is when we're analyzing data from a computer simulation. We can know we're using the correct (or incorrect) model because we wrote the program that generated the data!

But generally our model is a simplified representation (sometimes called an idealization) of a more complicated reality. In the context of the model, logic is infallible, but that may not translate back to the real world.

The apparent certainty of logic may encourage polarized thinking—a trap I've fallen into on, ahem, one or two occasions. When all propositions are simply TRUE or FALSE, everything is so easy, so tidy. But so misleading.

I'm not discounting the value of logic, but I am pointing out that we have to be very careful not to make it something it's not.

Questioning our models

Rene Descartes famously questioned all of his assumptions, arguing that:

If you would be a real seeker after truth, it is necessary that at least once in your life you doubt, as far as possible, all things.

Finally, he was left with no beliefs that he felt he could justify but the fact of his own existence. Although this approach seems a bit extreme, the idea that we should subject our models to careful examination is of paramount importance.

In order to honestly critique our models, we need to "think outside the box". A model is indeed a kind of box, and often an opaque one at that. We can grow very comfortable inside our models, to the point where we can scarcely conceive of another approach.

In the field of statistics, there is a large literature on model selection. The simplest case is that of nested models, where one model is an extension of another. But, writes Malcolm Forster:

Models belonging to different theories, across a revolutionary divide, are usually non-nested. A typical example involves the comparison of Copernican and Ptolemaic models of planetary motion. It is not possible to obtain a sun-centered model from a earth-centered model by adding circles. Cases like this are the most puzzling, especially with respect to the role of simplicity.

This is closely related to what Thomas Kuhn called the incommensurability of scientific paradigms.

The fine print

Of course there's more to say. After all, it's complicated ...

First, I want to point out that I'm using the term "models" in a very general sense. I'm including scientific models, theories, paradigms, mathematical models, statistical models, mental models, metaphors, assumptions, beliefs, moral/ethical models, and even theological models. And probably a bunch of other types of models I haven't thought of. Some people use the term in a much narrower sense. But my point here has been that several issues around simplicity are important for many different kinds of models.

Models serve a variety of purposes. Models can be descriptive, explanatory, causal, predictive, or normative. (And again, I'm sure this list should be longer.) It would be interesting to examine how simplicity plays a role in these different cases. Of course I haven't really defined simplicity, and that's a whole other area.

Finally, here's an entertaining list of
rules of mathematical modeling.

In memoriam: Ram Myers

Goodbye, dear friend.

I learned of Ram's death on Tuesday evening, from the mother of Ram's wife (Rita, who is also a dear friend).

She added that Rita has requested that people not try to contact her.

It's now early Wednesday morning, and I am going to bed.

Unfloppable?

As I noted the other day, a large part of the media has obediently lined up to proclaim the glories of Apple's "revolutionary" new iPhone. It seems that simply putting an i at the beginning of the name confers the blessing of the zeitgeist. With such an auspicious title, surely the iPhone couldn't flop. It's unsinkable unthinkable!

Or is it? Check out Kirk Sato's 10 reasons IPhone is going to be a Flop. (He also links to a hilarious Stephen Colbert segment.)

A purportedly potent rodent

And another thing: the iPhone has no buttons. Reminds me of another Apple product: the Mighty Mouse, which came with my iMac. The advertising copy is seductive:

Thanks to a smooth top shell with touch-sensitive technology beneath, Mighty Mouse allows you to right click without a right button. Capacitive sensors under Mighty Mouse’s seamless top shell detect where your fingers are and predict your clicking intentions, so you don’t need two buttons — just two fingers.

A smooth shill for a smooth shell. But--gasp!--Apple doesn't always get things right. For a more reality-based assessment of the purportedly potent rodent, check out the Wikipedia entry. The "Criticisms" section lists quite a number of purported shortcomings. I can personally corroborate these ones:

• Right clicks can be difficult. The fingers must be lifted completely off the left side of the mouse for a right click to work.
• The scroll ball is sensitive to dirt, and difficult to clean because it is not removable. It is often rendered inoperable and irreparable after only a few months of use.
• The squeeze buttons do not provide much tactile feedback and can be awkward to reach.

The issue of tactile feedback (or lack thereof) brings us back to the iPhone:

How can you dial the iPhone without looking at it? How can you reach in your pocket and press “1” for voicemail? How can you orient yourself with the interface without seeing it? With a traditional phone or device with buttons you can feel your way around it. You can find the bumps, the humps, the cut lines, the shapes, the sizes. You can find your way around in the dark. Not with the iPhone.

(Posted by Jason on 37signals).

Floptics

In the end, a flop may be in the eye of the beholder. And prediction without precision deserves derision. (That was original, by the way.) Whereas Nostradamus could get away with making vague and poetical prophecies, it behoves me as a scientist to make a falsifiable prediction. I haven't quite formulated this yet, but it seems to me that:

• Apple has obviously sunk a pile of cash into developing and marketing the iPhone.
• It's not clear that they'll recoup this investment.
• Nor is it clear that they'll still be selling phones three years from now.

This was the sort of thing I had in mind when I made my initial prediction (flop cit).

I'll conclude this post by taking a page out of Nostradamus' book:

I do but make bold to predict (not that I guarantee the slightest thing at all), thanks to my researches and the consideration of what judicial Astrology promises me and sometimes gives me to know, principally in the form of warnings, so that folk may know that with which the celestial stars do threaten them. Not that I am foolish enough to pretend to be a prophet.

(Open letter to Privy Councillor [later Chancellor] Birague, 15 June 1566.)