An ORnery problem

Some time ago, I wrote about missing values and how they complicate the life of an applied statistician. A particularly tricky case concerns logical variables, and I give a more detailed explanation here.

Suppose X is a variable representing whether a person is at risk for developing type-2 diabetes. Two known risk factors are (A) being older and (B) being overweight. If we had a database containing the age and weight of each person in a group, we could compute X using the following logical expression:

X = A OR B.

(X, A, and B are known as logical variables, and they each take values TRUE or FALSE according to whether the corresponding condition holds.) But what happens if some ages and weights are missing from the database? Fortunately statistical software packages like R and SPSS have built-in rules that will correctly evaluate the logical expression, even if A or B (or both) are missing. The complete truth table is as follows (where T means TRUE, F means FALSE, and a dot means missing):

Note that if A is FALSE and B is missing, the result is missing. That makes sense because if the actual value of B were TRUE, the result would be TRUE, but if the actual value of B were FALSE, the result would be FALSE. Thus it is not possible to say what the value of X is.

The trouble is, this logic can sometimes be perverse. Suppose X instead represents whether a patient tests positive for infection with a certain virus. But there may be two different blood tests (A and B), and patients may receive one or the other, or perhaps both. Suppose that if any of the tests is positive, the patient will be considered to be infected. The logical variables A and B take the values TRUE, FALSE, or missing according to whether the corresponding tests were positive, negative, or simply not performed. Shouldn't the logical expression X = A OR B handle this situation correctly? Unfortunately not. Suppose only one test was performed, and it was negative. Then the truth table shows that X will be missing, even though the patient tested negative!

Why does the logical expression handle missing values the way we want in the first case, but fail to do so in the second? The answer is that in the first case a missing value represents the fact that the age or weight of a given person is not available, whereas in the second case, when a test outcome is missing from the database, it means that no test was performed, thus the variable representing the outcome is not applicable. Another common case of variables that are not applicable occurs with data representing observations on multiple occasions. For example, suppose a database records whether hotel guests eat at the hotel restaurant on the first day of their stay (EAT1), the second day (EAT2), or the third day (EAT3). Some guests stay for just one day, while others stay longer. The database may look like this:

This is an example of a ragged array, and as with the blood test, the issue is that the denominator (the number of tests performed, or the number of days a guest stays at the hotel) varies. To determine whether a guest ate at the hotel restaurant at least once (which we will represent by the logical variable EAT), we might try:

EAT = EAT1 OR EAT2 OR EAT3.

Unfortunately, as with the blood test example, this can fail when there are missing values. Guest number 4 in the table above stayed just one day at the hotel and did not eat at the restaurant, so EAT should be FALSE, but the expression above gives a missing value.

Workaround in R

In R, the vertical bar operator | represents OR, and missing values are represented by NA. For the diabetes example, the following behaviour is just what we want:

> FALSE | NA
[1] NA

In other words, when a person does not have one of the risk factors, but we don't know about the other one, then we don't know if the person is at risk. But for the blood test example, we need to use the following code:

> sum(FALSE,NA,na.rm=TRUE)>0
[1] FALSE

The sum function adds up logical values by treating TRUE as 1 and FALSE as zero. If the sum of the logical values is greater than zero, then at least one of the values must have been TRUE. Setting na.rm=TRUE tells sum to ignore missing values.

Workaround in SPSS

The situation is much the same in SPSS. For the diabetes example, the following works:

COMPUTE X = A OR B.
EXECUTE.

But for the blood test example, we need to use:

COMPUTE X = SUM(A,B)>0.
EXECUTE.

Note that the SPSS function SUM ignores missing values.

Missing value mistakes

The hard part, of course, is thinking through how the missing values in a given situation should be handled. I suspect that this issue has resulted in countless errors in data analyses. Proceed with caution: a miss is as good as a mile!

Why do we overinterpret study findings?

MSNBC recently reported that a new study suggests "U.S. states whose residents have more conservative religious beliefs on average tend to have higher rates of teenagers giving birth". (I learned of this on Rationally Speaking.) The study itself is Open Access, so all the details are freely available. The scatterplot illustrates the strong association the authors found. Now, the authors were reasonably cautious in how they interpreted their findings. The trouble is, the general public may not be.

A common error is to conclude the study shows that religiosity causes higher teen birth rates. But correlation does not imply causation. It could be that higher teen birth rates cause religiosity. Or perhaps a third, unidentified factor causes both.

But isn't the strength of association still impressive? It is. But what if, as I just suggested, there are other variables involved? Such confounding variables (or confounders, as they are commonly known) can wreak havoc on this sort of analysis. Indeed, the authors of the study did adjust for median household income and abortion rate (both at the state level). But it is possible that other confounders are lurking. And unfortunately, we tend to forget entirely about the possibility of confounders when we hear about study findings.

Another error is to conclude that the findings directly apply to individuals. Here I will quote the authors directly:

We would like to emphasize that we are not attempting to use associations between teen birth rate and religiosity, using data aggregated at the state level, to make inferences at the individual level. It would be a statistical and logical error to infer from our results, “Religious teens get pregnant more often.” Such an inference would be an example of the ecological fallacy ... The associations we report could still be obtained if, hypothetically, religiosity in communities had an effect of discouraging contraceptive use in the whole community, including the nonreligious teens there, and only the nonreligious teens became pregnant. Or, to create a different imaginary scenario, the results could be obtained if religious parents discouraged contraceptive use in their children, but only nonreligious offspring of such religious parents got pregnant. We create these scenarios simply to illustrate that our ecological correlations do not permit statements about individuals.

To err is human ...

My goal here has not been to criticize the authors of this study, nor the media. Rather, what I find remarkable is how such a simple statement—"states whose residents have more conservative religious beliefs on average tend to have higher rates of teenagers giving birth"—can be so easily misinterpreted, and in so many different ways! Does anyone know of any research about our tendency to overinterpret scientific findings? Of course, we'd probably overinterpet it.

Absence of evidence ...

In a valid deductive argument, the conclusions follow necessarily from the premises. This is a proof in the mathematical sense of the word. Provided we know that the premises are true, we can establish with complete certainty that the conclusions are true. For example, identifying a single unicorn would establish without a doubt that unicorns exist.

Unfortunately much of the time this type of certainty isn't possible. Consider another example from the realm of mythology: weapons of mass destruction (WMD) in Iraq. Here's what Donald Rumsfeld had to say on the subject in 2002 (the boldface is my addition):

There's another way to phrase that and that is that the absence of evidence is not evidence of absence. ... Simply because you do not have evidence that something exists does not mean that you have evidence that it doesn't exist.

But surely hunting high and low for WMD month after month and not finding any (absence of evidence) supports the inference that there aren't any there (evidence of absence). Indeed, it turns out that the popular maxim cited by Rumsfeld is simply incorrect.

But didn't he have a point? Absolutely: failing to prove that something exists does not prove that it does not exist. Or, in the words of the English writer William Cowper (1731-1800):

Absence of proof is not proof of absence

Compare this with the version invoked by Rumsfeld:

Absence of evidence is not evidence of absence

The originator of this maxim seems to be the cosmologist Martin Rees, although it has been attributed to many others, including Carl Sagan. By substituting the word evidence for proof it makes a much stronger (and invalid) claim. Evidence, after all, is often uncertain. If I look outside and see that the ground is wet, that is evidence that it has been raining. But perhaps my neighbour was watering her flowers. Seeing someone walk by with an umbrella folded under their arm might strengthen my evidence for the rain hypothesis, but perhaps they are anticipating rain later on. In general, evidence can support an inference, but it won't necessarily prove it. And that's where Rees's formulation of the maxim falls down.

Black and white thinking about evidence

When evidence is construed as being certainty, we get into all kinds of trouble. This is how Rumsfeld turned a simple truism (no WMDs have been found, but they might still be) into a puzzle of obfuscation (absence of evidence is not evidence of absence).

But Rumsfeld is not the only one. As I noted recently, the term "no evidence" is commonly used to describe situations where an effect is not found to be statistically significant. Now statisticians are wary of people concluding that a lack of statistical significance implies that there is "no effect". (It might be, for example, that the sample size was inadequate.) Hence, it is not at all uncommon for statisticians to declare that absence of evidence is not evidence of absence! As Kim Øyhus has pointed out, even the American Statistical Association buys into it, as the t-shirt they sell attests.

Of course statisticians know well that uncertainty isn't easy to think about or communicate to others. So why have we fallen into this trap?

Well, part of the reason may be philosophical. Statistical reasoning is inescapably inductive—it does not guarantee certainty. Philosophers have been worrying about what is called the problem of induction for a very long time. David Hume (1711-1776) challenged the logical foundations of induction, and ever since, philosophers have sought a way around the problem. The reigning "solution" is known as the hypothetico-deductive method, developed by philosopher of science Karl Popper (1902-1994). Popper argued that induction in science could be avoided by proposing a hypothesis and then seeking evidence that would either prove the hypothesis wrong ("falsify" it) or fail to do so. This is very similar to the frequentist statistical hypothesis testing framework that developed from the work of Fisher, Neyman, and Pearson. Unfortunately, it lends itself to black-and-white thinking. A hypothesis is either proven wrong or it isn't. There's no grey zone.

Popper's formulation, in particular, buries the uncertainty completely, construing the reasoning as entirely deductive. Suppose, for example, that a new biochemical theory predicts that a certain drug will shorten the duration of an illness, whereas the older theory does not. Now duration of illness depends on numerous factors, including differences in patients' immune systems, and we expect to see variation above and beyond any differences due to the drug. A clinical trial may demonstrate that the average duration of illness for patients who are randomly assigned the drug is shorter than that for patients who receive placebo, and that this difference is statistically significant at the 0.05 level. Has the older theory been proven incorrect? Not with absolute certainty. The evidence against it may be strong but it is possible that this is a "type-I" error—rejecting the null hypothesis even though it is true. Indeed, because of the way the statistical test has been designed, when the null hypothesis is true we expect to see such errors 5% of the time. The companion to the type-I error is the type-II error—failing to reject the null hypothesis even though it is false.

Pretending that type-I and -II errors don't exist is wishful thinking. Just as diagnostic tests produce false positives and false negatives, statistical hypothesis tests can give the wrong answer. The point is that we can study and control the error rates and make inferences while acknowledging their limitations.

It suited Donald Rumsfeld's purposes to be fuzzy about the distinction between evidence and proof. It doesn't suit ours.

Update 03-Jun-2009: I had originally attributed the maxim "Absence of evidence is not evidence of absence" to Carl Sagan in his 1995 book The Demon-Haunted World." Apparently however, the originator was cosmologist Martin Rees. There is reference to it in the proceedings of a 1972 symposium titled Life Beyond Earth & The Mind of Man [pdf], jointly sponsored by Boston University and NASA. In his introductory remarks, the chair, Richard Berendzen stated:

A generation ago almost all scientists would have argued, often "ex cathedra," that there probably is no other life in the universe beside what we know here on Earth. But as Martin Rees, the cosmologist, has succinctly put it, "absence of evidence is not evidence of absence." Beyond that, in the last decade or so the evidence, albeit circumstantial, has become large indeed, so large, in fact, that today many scientists, probably the majority, are convinced that extraterrestrial life surely must exist and possibly in enormous abdundance.

(The boldface is mine.) Note that Carl Sagan was one of the panelists at the symposium.

Something fishy about "no evidence"

Searching Google for "no evidence" yields "about 21,300,000" hits. It seems we're keen to deny that there is any empirical support for countless different claims. For example, scientificblogging.com reports that there is "No Evidence For Fish Oil Benefit In Arrhythmias" based on a systematic review just published in BMJ. (Full disclosure: I have previously participated in research on omega-3 fatty acids, however I have no related financial interests.) What does the review itself say?

This is the first systematic review attempting to evaluate whether the protective mechanism of fish oil supplementation is related to a reduction of arrhythmic episodes determined either by a reduction in implantable cardiac defibrillator interventions or a reduction in sudden cardiac death. We found a neutral effect on these two outcomes. The confidence intervals for these outcomes were wide and a beneficial effect up to a 45-48% relative risk reduction cannot be excluded.

To better appreciate this, here's their Figure 2:
Note that of the three studies that looked at the proportion of implanted defibrillators that were triggered, one showed a statistically significant effect in favour of fish oil, and the other two did not show statistically significant effects (one favoured placebo and the other favoured fish oil). Six studies looked at sudden cardiac death in patients taking fish oil compared to those taking placebo. Only one was statistically significant, and it favoured fish oil. Of the five studies that did not show statistically significant effects, two favoured fish oil, and three favoured placebo.

The diamond shapes in the figure show the pooled estimates with their 95% confidence intervals: in each case the diamond overlaps an odds ratio of 1, indicating that the overall effect is not statistically significant. And when there's a non-statistically significant effect, it is common practice to say there is "no evidence". But that can be very misleading! After all, two of the individual studies did show a significant benefit of fish oil. So what's going on? Well, for starters, there's some indication of heterogeneity between the studies (particularly in the case of the defibrillator studies). But it also seems that more large studies are needed: a good deal of the variation in results between the studies may simply be due to the play of chance. Quite substantial benefits of fish oil are entirely plausible: relative risk reductions of as much as 45-48%!

What is "no evidence"?

Consider the figure below:

At the bottom there is a gray axis line with tick marks and a vertical gray line indicating the "null" value (where there is no preference one way or the other). The blue line with arrows at each end represents an infinitely wide confidence interval. This is the most straightforward representation of "no evidence": there is simply no empirical information to indicate what the true effect might be.

But suppose we have a very small sample, that is, one that provides almost no empirical evidence. The figure might become:

The only difference is the blue dot on the confidence interval just a bit to the right of the null line. It represents the point estimate based on a very small amount of empirical information. Of course it could equally well have been on the left hand side (or perhaps directly on the null line). Regardless, the confidence interval is still very wide, so very little can be said about the true effect. With such a wide confidence interval, the location of the point estimate is almost irrelevant.

Finally, suppose that a reasonably large sample is available:

I have left the point estimate at the same place. The confidence interval no longer has arrows on either end and is relatively narrow. However it still overlaps the null line. That means the estimate is not statistically significant. Sometimes this sort of situation is described as showing "no evidence of an effect". But as I noted above, that's quite misleading language. In fact, what this situation shows is indeed evidence—evidence that any effect likely has a magnitude of no more than two tick marks (whatever they represent) on the right hand side of the null line or a magnitude of no more than about a half a tick mark on the left hand side of the null line.

But here's the tricky part: what do those ticks represent? Suppose the axis represents annual cost savings that might result from implementing a certain type of federal government program. If each tick mark represents $1000, then we have estimated that the program will cost at most $500 a year and save at most $2000 a year. In other words, the program has been shown to be effectively revenue neutral: the evidence suggests that the cost/cost-savings of the program will not be important. On the other hand, if each tick mark represents one million dollars, most of us would feel that the jury's just not in yet. A possible cost of $500 a year is a drop in the bucket, but $500,000 a year is something else entirely.

In a way, money is the easiest measure to evaluate like this. Things like safety are much harder. For example, if the evidence suggests that a certain chemical may increase the rate of certain types of cancer, but the findings are not statistically significant, what can we conclude? Can the manufacturer claim that there's "no evidence" the chemical is harmful? Can health activists claim that there's "no evidence" the chemical is safe?

I would argue that the term "no evidence" is inappropriate in either case. The underlying questions remain: what is required in order to conclude that a chemical is harmful or that it is safe? Ultimately there's no getting around the issue of how large a difference (in, for example, cancer rates) has to be in order to be considered important. And that's a rather uncomfortable question.

StatLinks

The Internet makes it possible to link a dispersed community of common interest. Now there are a number of blogs that focus entirely or in part on Statistics, but they seem not to be well connected.

So I've just set up a social bookmarking website just for applied statistics, data analysis, and visualization. It's called StatLinks.

It lists links that users submit, and allows other users to vote on their relevance. Links are listed in order of popularity (or in chronological order, if you prefer).

I encourage people to visit StatLinks, to submit links that are likely to be of interest, and to pass the word! I've put a few links in to get things started. (Hat tip to Slinkset whose technology made it a breeze to set this up.)

Is it fair not to share?

Over at Adventures in Ethics and Science there's a very interesting post and follow-up comments on whether researchers should share data. It's based on a recent New York Times article by biostatistician Andrew Vickers (Cancer Data? Sorry, can't have it).

Paint by numbers

The map on the left labels countries as free (green), partly free (peach), and not free (red) for the year 2006. The classification is from Freedom House, a primarily US-government-funded organization. Each country is scored on political liberty and civil liberty; the combined average of these scores determines how the country is classified. Cuba, for example, receives the poorest rating on both political and civil liberties, and is thus categorized as "not free".

The Wikipedia entry for democracy lists two other measures of democracy, one from the Polity IV project and the Democracy Index from The Economist. Each measure has its own scheme for measuring and weighting different characteristics felt to characterize democracies. Since the different measures assess many of the same things, it is not surprising that they show some agreement.

But democracy is not a simple thing and it is far from clear which characteristics matter and how best to weight them. For example, does a free press count more than an independent judiciary? If so, by how much? (And presumably this depends on how "free" and how "independent" they are.) What about non-traditional characteristics that may be important measures of democracy? For example, should voter turnout be factored in? What about incarceration rate? Or media concentration? Or universal health care?

Although there may be some value in overall measures of democracy, individual characteristics still need to be examined and put in context. When we do that, we may find that the global canvas no longer looks like a paint by numbers kit.

Update: 05Mar2008

No sooner had I put up this post than I saw this Adbusters press release relating to media concentration and democracy:

Adbusters Demands Canwest, the CBC and the CRTC Stop Blocking Citizen-Produced Advertising

On Monday, February 18, Adbusters lost its court battle against two of Canada's television networks that refused to sell airtime for its commercials. Adbusters claimed the CBC and Canwest Global had violated its right to free speech under the Canadian Charter of Rights and Freedoms by refusing to sell air time, but the court decided that the Charter does not apply to private corporations.

"It's outrageous that the fast food, oil and automobile industries can buy as much TV time as they want in order to promote their agendas, but citizens are not allowed to talk back," said Adbusters Editor-in-Chief Kalle Lasn in response to the ruling. "Canadian democracy will not work properly until we the people have the same right to buy airtime as corporations do."

The rejected Adbusters ads pointed out that over 50 percent of the calories in a Big Mac come from fat, called for an end to the age of the automobile, and promoted Buy Nothing Day. While Court Justice William Ehrcke ruled that private broadcasters have the right to run whatever ads they like, Adbusters feels the case raises some troubling questions.

Firstly, why are Canwest and the CBC selling as much time as they possibly can to corporations, while fighting expensive legal actions to keep citizen-produced messages off the air? Why does the CBC call itself "Canada's Public Broadcaster" if they won't sell airtime to citizens?

Secondly, why is the CRTC not standing up for public access? When they grant licences to broadcasters, why is the right of Canadian citizens to access their own "public" airwaves not being guaranteed? Thirdly, why is our freedom of speech being suppressed? Why can corporations buy airtime while citizens cannot? Why doesn't the Canadian Charter apply to the most powerful social communications medium of our age - television?

"This case goes to the very heart of what our democracy is all about," says Lasn. "A healthy society allows its citizens to walk into their local TV stations and buy airtime under the same rules and conditions that corporations do. Adbusters has been given 30 days to challenge the ruling. This legal battle for media democracy will go on."

To talk to Kalle Lasn, or Ryan Dalziel, our lawyer, about the case please contact Lauren Bercovitch (lauren@adbusters.org)

EDITOR'S NOTES

For more information about Adbusters and the global media democracy movement visit www.mediacarta.org and www.adbusters.org

[1] Canadian Media facts:

Three corporations (CanWest, Quebecor and Torstar) control 70 per cent of the country's daily newspaper circulation.

Five major media acquisitions in Canada have been approved by CRTC in the past year: CHUM was purchased by CTVglobemedia for $1.4 billion, which then sold five CityTV stations to Rogers Communications for $375 million; CanWest purchased Alliance Atlantis for $2.3 billion; Astral Media bought Standard Broadcasting for $1.2 billion; and Quebecor bought the Osprey Media newspaper chain for $414 million.

[2] Facts about Media Democracy:

More than 30,000 people have signed the Media Carta www.mediacarta.org, to voice their concerns about the way information is distributed in our society.

In the past year, a growing number of grassroots media activist groups have been formed in Canada to express a dissatisfaction with the continued consolidation of the country's media:

DemocraticMedia.org

MediaReform.ca

MediaDemocracy.ca

Data and development

Here's a fascinating talk by Hans Rosling about international health and development. His presentation reminds me of Al Gore's An Inconvenient Truth.

Thoughts?

LOLstats

The princess and the outlier

In my continuing effort to eff the ineffable (consciousness), I today stumbled on an article by Jaron Lanier with the intriguing title "Death: The skeleton key of consciousness studies?" It's written in an entertaining manner with little in the way of technical jargon. Lanier makes some interesting points, but what struck me was the following piece near the beginning:

There is a popular story about a princess who complains that she cannot sleep comfortably because of a single pea buried under layers of mattresses. That pea is consciousness in the sciences.

To consider consciousness by itself is entirely undemanding. It is a pea. There is nothing to describe. An attempt to account for it in context, however, forces the construction of ever shifting, elaborate adventures of thought.

What a temptation it is to dispose of this erratic data point. That is what any first year student of statistics would be taught to do.

Excuse me? I was enjoying the metaphor until that last bit!

At this point I'm tempted to launch into an extended discussion of statistical approaches to outliers. Or an outraged defense of statisticians against the notion that we teach first year students to casually discard data points that seem aberrant. But I think I'll put it on my to-blog list. That's one more pea under my matress!

Things that (probably) don't exist

A recent article by philosopher Steven Hales is titled "You Can Prove a Negative" (a slightly different version of the article is available as a pdf file). Hales argues that the "principle of folk logic" saying you can't prove a negative is just plain wrong.

He points out that "any claim can be expressed as a negative, thanks to the rule of double negation." So it's easy to come up with examples of proving a negative. Hales goes on to say that "Some people seem to think that you can’t prove a specific sort of negative claim, namely that a thing does not exist." He counters this with an example of a valid proof that something doesn't exist:

1. If unicorns had existed, then there is evidence in the fossil record.
2. There is no evidence of unicorns in the fossil record.
3. Therefore, unicorns never existed.

Of course, the difficulty here is with the truth of the premises (1 and 2). In particular, it could be that we just haven't found unicorn fossils yet. Or perhaps, unicorns don't leave a fossil trace. Deductive arguments are so neat and tidy we may forget about what's been swept under the carpet: the truth (or otherwise) of the premises.

Finally Hales grasps the nettle:

Maybe people mean that no inductive argument will conclusively, indubitably prove a negative proposition beyond all shadow of a doubt. For example, suppose someone argues that we’ve scoured the world for Bigfoot, found no credible evidence of Bigfoot’s existence, and therefore there is no Bigfoot. A classic inductive argument. A Sasquatch defender can always rejoin that Bigfoot is reclusive, and might just be hiding in that next stand of trees. You can’t prove he’s not! (until the search of that tree stand comes up empty too).

And now we come to the heart of the matter:

The problem here isn’t that inductive arguments won’t give us certainty about negative claims (like the nonexistence of Bigfoot), but that inductive arguments won’t give us certainty about anything at all, positive or negative. All observed swans are white, therefore all swans are white looked like a pretty good inductive argument until black swans were discovered in Australia.

Well, hold on just a moment. We were talking about "a specific sort of negative claim, namely that a thing does not exist". And the swan argument hasn't been written that way. If we do write it that way, we get the inductive argument no observed swans are black, therefore all swans are non-black. So non-existence claims based on observation are uncertain.

But what about existence claims based on observation? Well, you only have to see one black swan too conclude that not all swans are white, and this inference is certain because it's deductive. (This is, of course, provided that we can trust that what we've seen really is a swan, and it really is black, and that we didn't just imagine the whole thing. There are some important issues here, but taking this too far can lead to radical skepticism, which is unproductive.)

My point is that when it comes to using observational evidence to argue for existence (a positive claim) or non-existence (a negative claim), you can't prove a negative, whereas you can prove a positive. (Here I'm using "prove" to mean "establish with certainty".) So, in this sense, I disagree with Hales. And I think that this is what people typically mean when they state that "you can't prove a negative". I also think that the imbalance in the difficulty of demonstrating non-existence compared to existence is a strong argument that the burden of proof should be on those who claim the existence of something.

I agree with Hales, however, in his defense of induction:

The very nature of an inductive argument is to make a conclusion probable, but not certain, given the truth of the premises. That's just what an inductive argument is. We’d better not dismiss induction because we’re not getting certainty out of it, though.

I believe we all crave certainty, but it's in pretty short supply—caveat emptor.

If we weren't so terrified of uncertainty, we might make much better decisions. When it comes to things that can be quantified, the field of statistics offers some very useful tools for dealing with uncertainty. Suppose, for example, we're trying to determine whether all swans are white. If we sample, at random, 100 swans, and each of them is white, then a very useful approximation, the "Rule of Three" tells us that we can have 95% confidence that the true proportion of non-white swans is less than 3/100 or 3%. Suppose we continue sampling swans and they stubbornly continue to be white. Having sampled 10,000 white swans, we can now have 95% confidence that the true proportion is less than 3/10,000 or 0.03%.

The notion of "95% confidence" can be made precise (but I won't get into the details here). It's also noteworthy that there are Bayesian analogues to the Rule of Three. Details are in Jovanovic and Levy, A Look at the Rule of Three, 1997, The American Statistican, 51: 137-139.

Unfortunately, there's a major difficulty in the application of the Rule of Three to the swan example: the assumption that the swans are randomly sampled! It turns out that the black swans were hiding out in Australia. But there's a message here: non-random samples can give very misleading information. That's one reason why anecdotal evidence is treated so skeptically by scientists.

For an atheist perspective on the "you can't prove a negative" idea, see here. And here's a page on burden of proof relating to claims of existence, from philosopher Philip Pecorino.

Update 12Dec2007: I sent a link to this post to Professor Hales and he kindly replied:

You write that you only have to see one black swan to know that not all swans are white, and that “this inference is certain because it is deductive.” But wait—the argument I gave about unicorns was also deductive, and you dismissed that as proving its conclusion. Therefore you can’t hold that the conclusion of your swan argument is certain because the argument form is deductive. If the conclusion of the swan argument is certain, then it is for some other reason. I suspect that you think it is certain because you are convinced of your premise that we have seen black swans. Of course, I’m rather convinced of my premises that if unicorns had existed, then there is evidence in the fossil record, and that there is no evidence of unicorns in the fossil record. Before you rejoin that we could find out that we are mistaken about the fossil record (as we would discover if we locate a unicorn skeleton), let me point out that we could also be mistaken about observing black swans. Maybe upon further study we’ll find out that they aren’t swans at all, but are merely related to swans. Or we could discover that they were phony, dyed white swans prepared to fool naïve naturalists. Or we might show that other even more skeptical hypotheses are true (mass hallucinations, dreaming, etc.). The real problem, as I see it, is your equation of proof with certainty. Most epistemologists don’t think we are certain of anything outside of logic, mathematics, and other things known a priori. There is always the possibility of error. But that doesn’t mean that we can’t prove things in some reasonable, real-world sense of prove.”

In memoriam: Ram Myers

Goodbye, dear friend.

I learned of Ram's death on Tuesday evening, from the mother of Ram's wife (Rita, who is also a dear friend).

She added that Rita has requested that people not try to contact her.

It's now early Wednesday morning, and I am going to bed.