Numbers for Words People

My family and I were driving back home one evening, when my wife, Terri, asked our son, a Grade Three student at the time, “Dimitri, which friends do you play with?” Dimitri answered with a few names, including “Ashley”. 

Terri: “Boy Ashley or girl Ashley?”

Before he could answer, I interjected: “Don’t be silly! I have taught many Ashleys over the years. They were all girls.”

Dimitri: “That’s because you teach at a girls’ school!”

The now adult Dimitri would have said, “Dad, you are looking at a non-representative sample of the population!”

This post is about sampling, both good and bad, and one phenomenon that causes people to draw conclusions from the wrong sample, known as the survivorship bias.

Representative Samples

Sticking with the above anecdote, let’s say we wanted to know which sex was more likely to be given the name Ashley. We certainly shouldn’t ask me, for the reason pointed out by my son. We would need to go to a co-educational school and ask the administrators to give us their breakdown of the name according to sex.

Would we then know whether Ashley was more common among girls or among boys? Not necessarily. One school is too small a sample. The area in which the school is located may have special characteristics, such as a preponderance of families from a particular ethnic background. You won’t find many “Ziads” in an area where hardly anyone comes from the Middle East.

Since we live in the age of technology, why not post a poll on a website and give people a week to answer it? Online polls have their problems: Only those interested in your question will answer, it is hard to know that each respondent is answering once, and you could have a campaign by those who feel strongly about a cause that ends up misrepresenting reality. It is usually better to poll people: Go to them yourself, ask the question and write down their answer.

In our case, we need to poll schools from different parts of the state or country.

When your friends are your sample

Rita is the CEO of the Fictional Bank of this Blog (the famous FBB). Her sister, Leah is a social worker and works at the Fictional Blog Community Centre. Rita and Leah swim in different political waters. The two sisters are meeting for coffee and discussing the upcoming elections in Fictional Blog Land.

Rita: “Everyone I talk to tells me that they’ll be voting right-of-centre”.

Leah: “Really? That’s not what I’ve been hearing.”

If you have ever been surprised by an election result, it is probably a sampling error. Your expectations were based on a sample of people, namely your friends and family, who did not represent the entire population. Such a sample is said to be “non-representative”.

When the sample comes to you

In World War II, the US military noticed that many of its airplanes were returning with bullet holes to the fuselage, tail and wings. They wanted to reinforce these areas with armour. The tradeoff was going to be that the extra armour would mean slower planes with a shorter fuel range. They had the good sense to consult the “Statistical Research Group” at Columbia University.

Abraham Wald, a mathematician at the SRG, advised the military to reinforce the engines and cockpit instead! You see, they were judging the need for reinforcement by the planes that had returned, not by those whose damage had prevented them from returning!

This is a classic example of what is known as “survivorship bias”. The planes that survived were the ones that the military had observed and decided to base its decisions on. The planes that had fallen hadn’t made the sample!

Survivorship bias in business

Trevor is driven to succeed in his career. He is 19, with one year of college behind him. As an avid reader of biographies, he has noticed a pattern: Bill Gates, Steve Jobs, Mark Zuckerberg and others had all dropped out of college to build enormously successful businesses. Those immensely successful people were his “sample”. Is that a representative sample?

To answer this question, we would need to look at a few thousand people who dropped out of college to build a business, and calculate their success rate. Trevor is looking at the “survivors”, those for whom the strategy has worked out. No one writes biographies of failed entrepreneurs.

Conclusion

Much can be said about the importance and difficulties of picking a representative sample before setting out to conduct a poll or run an experiment. A bad sample can lead a business to go after the wrong market, a politician to misjudge their constituents’ priorities or a military to reinforce the wrong part of its planes. Medical trials conducted, exclusively, on male subjects have resulted in misdiagnoses and delayed treatment for women.

The next time a Youtuber tells you to take a supplement or adopt a health routine, based on a study, ask yourself: Who made up the sample and how large was it? 

---

If you enjoyed reading this, take a look at this article about another aspect of statistics, namely correlation. Here’s a good video on survivorship bias. Finally, let me know in the comments whether you want me to cover any other topics related to statistics.