Monthly Archives: September 2009

Statistically Speaking – The Lottery Fallacy

Have you ever thought about someone, then the phone rings and THEY are on the other end? Ever had a dream that something would happen and then it did a few days later? Are you somehow the cause of any of these events? Many would say yes, though no evidence exists to say that could ever be the case. What’s at work here is probability. This intuitive lack of understanding of statistics that I will elaborate on here.

Statisticians tell us we are virtually more likely to be hit by lightning than win the lottery. An event such as lightning striking some poor sod happens everyday despite the odds of it being incredibly slim. The fact is, it does happen because given enough lightning strikes (there are thousands every day) some will just happen to be in line with a person.

Remember this analogy when someone uses the argument from personal incredulity (I can’t believe it therefore it can’t be). Human beings have a poor innate sense of statistics. This fact alone is responsible for a good number of weird beliefs people hold. Science is in one sense a way of overcoming the flaws inherent in our perception that lead us to draw false cause and effect inferences.

The superstitious mammal

The religious and superstitious impulses thrive in conditions where we encounter something we can’t fathom – something so improbable it seemingly defies reason. This is where statistical fallacies begin. There are two flavours: fallacies arising from improbable events taking place; and statistics of small numbers.

An example of the first one often trotted out by religious apologists is that conditions are too perfect to have been created by chance. Therefore there must have been an intelligent force that created these conditions for us.

There are several fallacies in reasoning at work here. The major flaw being made is that if you look at some property of the universe in isolation, it can appear more miraculous than it actually is. What are the odds that the perfect conditions for life would arise on Earth? The probability is, of course, very small.

This however, is ignoring the fact that Earth is but one place in a universe filled with trillions of stars, many with which would have planets revolving around them. When you ask the correct question – What are the odds of conditions being perfect for life arising somewhere in the universe? The answer is of course very probable (we know this is a 100% probability because we are here!)

This is called the lottery fallacy – what are the odds that you will win the lottery? The answer is the same as the scenario above – very small (too miniscule to rely on winning it to be successful in life as many do). What are the odds that someone will win it?

What do most people say when they win? It’s a miracle! I’m so lucky! The real answer is neither luck nor miracles. Someone was going to win the lottery. The person who does win does so at random.

Random acts of randomness

But random doesn’t sit well with the human mammal. We make all kinds of inferences about causes and effects. We are so good at it, we can often ascribe effects to causes where no scientifically plausible link exists. As Bruce Hood says in his book Supersense Why We Believe the Unbelievable, our abiltiy to infer cause and effect allowed us to thrive in the environment better than our rivals.

The science shows that we are so good at creating causal links that it is innate in us to go beyond the evidence to infer causes that we cannot detect. In fact, given the choice, people would rather have some meaning than none at all, regardless if the created meaning is real or not. People, understandably, favour feeling certainty over uncertainty. Belief before doubt.

Given these tendencies inside all of us, it is hardly surprising that a many superstitious-based belief systems thrive today. Attributing supernatural agency as a solution to “how’d that happen” is written in our genes.

In a city of a million people, events of a 1/1000,000 probability occur all the time.

Statistics of small numbers

The second fallacy – the fallacy of statistics of small numbers relies on developing conclusions after only a few or even one experience. This has its advantages. Cognitive shortcuts are obviously more favourable to a mammal because that means less exposure and hey, the initial impression would have been enough in primitive humans. And so we form beliefs about the world based on a small subset of available experiences.

People do this all the time. A woman goes on five dates and meets five loser guys. The questions flow: “What’s wrong with me?” “Why can’t I find a good guy?” The conclusions are drawn: “There are no good guys”; “all the good ones are taken”; “I will never find the right guy”. Are any of these correct? After five dates, it would be impossible to know for sure.

Of course, if the woman approached this situation as a scientist, she would realise that 5 is a small number given the number of available guys. She might think: “Maybe if I meet 20 guys and the current situation holds, then maybe my intuitions about guys has more validity”. But would 20 be a representative sample of the whole population? Statistically significant? No.

Shoddy research

Shoddy researchers, particularly those with prior ideological leanings, love small studies. Small numbers can be incredibly deceptive. Many claims to reality, particularly in the alternative medicine camp, are made on the basis of small studies. Science works on the basis of well designed, large studies with statistically significant samples sizes. Even then, one study cannot be the basis to confirm something as real. Over time, the good ideas survive multiple well designed studies and we can say, with a high degree of probability that something has scientific validity.

How not to report on pseudoscience

A great example of false balance and incorrect reporting of fringe pseudoscientific claims cropped up recently in the Sunday Herald. In this story, Wet, cold forecast for 2010by Rebecca Lewis, we encounter some rather odd reporting about New Zealand’s self proclaimed weather expert, Ken Ring. Lewis begins her article with the following passage:

 “Ken Ring, a long-term forecaster with unorthodox methods but a surprisingly accurate track record, has predicted next year’s weather to be “disappointing”, with wet and cooler summer months, followed by a winter that lasts a month longer than this year, with record-breaking cold snaps.

For background, Ken Ring claims he can predict the weather, months and even years in advance, by looking at lunar cycles – a claim not supported by science.

There are a couple of things at work here. For one, Ken Ring’s own explanation for how he arrives at his predictions is plausible sounding enough for the layperson. He uses the same scientific terms as the scientists do. This is often enough for most people to buy into. Combine with a liberal sprinkling of confirmation bias and voila, a pseudoscience is born.

After the initial preamble, Lewis continues:

Ring’s methods of using the moon and tides to forecast well beyond the timelines of MetService or NIWA have raised eyebrows in the scientific community for years, but the Kiwi weather watcher has been largely on the mark in New Zealand and overseas.

The next few paragraphs are citations of positive hits attributed to Ken Ring in order to support his claims.

The question then must be asked: why are these positive reports uncompelling to the scientific community? The answer is quite simple – what Lewis is putting forward as evidence for Ken Ring’s claims is purely observational selection bias. This is where one starts with an idea and finds supporting evidence for it while simultaneously disregarding contrary evidence.

Observational selection bias is very much a default setting of the brain and is one of the main reasons why human beings are not purely rational, objective recorders of reality. This is also why we test ideas scientifically. It is a common ploy by believers and proponents of all forms of pseudoscience to cherry pick data that supports their beliefs. However, it is only when the missing data is employed — the negative results — that we can get a complete picture of the validity of claims.

Tests smoke out bad ideas

In the case of Ken Ring’s “weather by the moon” ideas, the negative evidence is shows that his claims are not valid. In the case of a news article, negative results aren’t nearly as sexy or newsworthy and so are disregarded.

A similar article to Rebecca Lewis could be written about psychics and the conclusion could be drawn that the particular psychic in question was legitimate. All you would need to do is cite times when the psychic made hits and disregard the bulk of negatives that provide evidence that the psychic is not in fact legitimate.

A simple controlled scientific test would be the solution.

  1. Take people at random and ask them to predict the weather 12 months in advance. This would be pure guesswork on the part of the participants.
  2. You could then compare their hits and misses with those of Ken Ring’s moon predictions.
  3. If there were something to Ken’s claims, we would see a sizeable statistical difference from the control group.

I predict that we wouldn’t see that but as a test of this kind is yet to be done, Ken’s own claim, that anecdotal reports show his predictions to have an 85% success rate, are premature and misleading. Anecdotal reports are not strong form of evidence and are subject to the very biases that go into confirming beliefs, not verifiable facts. A scientific test would filter these biases out.

“Despite a huge following, Weather Watch analyst Phil Duncan is sceptical about Ring’s theories, saying it is too difficult to predict weather more than a month out.”

Ah… the argument from popularity. By this comment, Rebecca Lewis seems to be implying that skepticism of Ring’s ideas isn’t justified because he has a lot of followers. So do many cults, homeopathy, acupuncture and HIV deniers.

The argument from popularity is not a very compelling one at all when you look at how many popular beliefs people hang on to despite overwhelming evidence to the contrary. The amount of people who believe something to be true is completely irrelevant. The scientific method is indifferent to what people think (just think, if science were dependent on opinion we wouldn’t call it science).

I suspect that most people who believe in Ken Ring’s “weather by the moon” material aren’t aware that their beliefs are underpinned purely by confirmation bias.

A related issue with the story is why the writer finds Ken Ring’s work compelling in the first place, given the scientific consensus that his ideas reside under the category of quackery. The fact neither NIWA nor the MetService endorse his “science” and use it to make their weather and climate forecasts is a revealing fact. But Lewis disregards this as strong reason for skepticism of Ken Ring’s claims and therefore ends up grandstanding for an unscientific notion.

Lesson? Be Skeptical of Simple Answers

The idea that something as complex as the weather can be predicted by purely looking at lunar cycles is overly simplistic and therefore invalid. The implausibility of predicting weather by this method becomes all the more pronounced when attempting to foretell what the weather will be months and even years out simply by the moon.

If the moon were a good indicator of weather then we could expect there to be some regularity in weather phenomena year-on-year. To an extent, we do observe this in the changing of the seasons, but this is climate, not the weather and these observations are related to the axial tilt relative to the sun.

What we do observe when looking at meteorological data is this – the weather is random and subject to many influences that vary from year to year, independent of lunar cycles. We also find that variability in weather events occurs every year and can’t accurately be predicted until the necessary forces shaping that weather is in the melting pot (sometimes this can only be seen days or even hours ahead out). This is because weather is a chaotic system and such systems are in constant flux that we can never know ahead of time what is actually going to happen.

If the hypothesis that the moon is an accurate predictor of weather, we should expect weather to correlate with lunar cycles. The fact we do not see such a correlation is falsification of the hypothesis put forward by Ken Ring.