Beating Bias to Win Wars
SimonSays #1 - Survivorship Bias, WWII stories, Creating Ben & Jerry's and what the Covid Crisis means for Capitalism, Globalization and Humanity.
Hi,
Welcome to SimonSays #1! If you’d like to sign up, you can do so here. Or just read on…
Until next week,
Stay safe!
Story Time
A few months ago, I was visiting a friend at Columbia University, climbing Morningside Drive on another freezing day in New York. Little did I know, that 75 years prior, this same area was deeply involved in the US war effort in Europe and the Pacific. Indeed, walking the same streets from July 1942 to September 1945, were “the most extraordinary group of statisticians ever organized, taking into account both number and quality”, the Statistical Research Group (SRG).
This was not your average research or consulting group. To start with, it handled pretty serious matters. As the group’s Direct of Research, Allen Wallis, later recounted:
The first project assigned to the SRG was to evaluate the comparative effectiveness of four 20 millimeter guns on the one hand and eight 50 caliber guns on the other as the armament of a fighter aircraft. This assignment had been preceded by two months of study of air warfare analysis and plane-to-plane fire control. This first assignment involved a study of the geometry and tactics of aerial combat, the probability of hitting, and the vulnerability of aircraft.
Second, war meant that they were working under the constant pressure to deliver, and further to convince the military personnel to employ their findings.
Third, war is unique in that it’s both an offensive and defensive exercise. Thus, if you’re improving your anti-aircraft accuracy, you want to make sure that your own aircraft wouldn’t end up more vulnerable if that technology were used by the enemy, and so on. The work on the probability of hitting aircraft therefore led to an assignment on the probability of anti-aircraft hits on a directly approaching bomber, which then led to an analysis on the use of shrapnel against directly approaching planes and in turn to a study of the vulnerability of American planes to anti-aircraft weapons.
So being a researcher in such context is a constant mental challenge. It’s like training by playing a chess game against yourself, hoping to beat an enemy who’s also been training by playing chess against themselves.
One of the problems they were confronted to regarded aircraft survivability. A Navy Officer wanted to know how the armor should be distributed on aircrafts to limit their vulnerability to enemy fire. He had brought data collected from a previous mission, which included the total number of aircraft sent, the amount that had returned and how many bullets each returning aircraft had received. For the latter, the officer had divided the plane into four parts, and measured the number of bullet holes in each part, as summarized in the table below.
If you could only armor one part, which one would you choose? If you don’t want to commit to one particular area, you could also establish what information you would need in order to make a better judgement. We’re more concerned about intuition here and decision-making than the actual result.
So, ready to continue?
Story has it that the Navy Officer had also brought with him the results of a study made by researchers at the Center of Naval Analyses, which suggested placing the armor on the fuselage, given the high number of enemy bullets it received. (This story’s veracity is questioned).
Yet, Wald’s approach was quite the opposite. Jordan Ellenberg’s book How Not to Be Wrong quotes Wald saying: “The armor, doesn’t go where the bullet holes are. It goes where the bullet holes aren’t: on the engines.” The idea being, that the engines weren’t necessarily being hit less, rather, aircraft hit in the engines were less likely to survive and thus not included in the data.
With that in mind, Wald started calculating the probability an aircraft would survive a hit in the different parts. The greater the probability of surviving a bullet in a particular part, the less that area needed protection.
He obtained the following results.
The table clearly demonstrates the vulnerability of the Engines, compared to the rest of the aircraft.
Wald also acknowledged that other factors might affect the probability of surviving a hit, such as the angle of attack, or the type of ammunition used. If you’re curious about his mathematical calculations, check out his demonstration or an analysis of his demonstration.
The ‘lesson’ from this story, would be to always look out for information that isn’t there, the one that hasn’t “survived”. Otherwise, you fall prey to Survivorship Bias.
Survivorship Bias could be considered a subset of Selection Bias. By only considering what’s there, and not what’s not, you’re basing your calculations on a non-representative sample, thus biasing your results.
The Statistical Research Group was dismantled at the end of the war, in 1945, but had a considerable influence on post-war statistics. SRG is also often attributed the creation of sequential analysis, an important method in statistical analysis, where the sample size is not fixed in advance. The team and especially its Director, Allen Wallis, later played a role, in the establishment of five University Departments of Statistics: Columbia, Stanford, Chicago, Harvard, and Rochester. One of its members, Milton Friedman, went on to obtain the Nobel Memorial Prize in Economic Sciences.
Coronabias
Abraham Wald’s aircraft survivability story is one of the most commonly cited with regards to survivorship bias. But survivorship bias is all around us. I’m sure you’ve already heard people say “Bakeries (or other shops) are such a good business, two of those opened in my neighbourhood this year”. This could be true, however, be sure to check how many bakeries have closed in the neighbourhood the same year. If as many close every year, then it turns out they might not be such good businesses.
The Coronavirus crisis also offers examples of Survivorship Bias. Indeed, countries were often ranked by number of confirmed cases. However, news outlets rarely mentioned the total number of tests conducted, or the criteria or ease to get tested in each country. Did Germany really have more cases of Coronavirus, while testing 200 000 people, than France who tested only half that amount? Interestingly, or rather tragically, in the case of the Coronavirus, the sole focus was placed on the number of dead, rather than on the survivors, an anti-survivorship bias?
There’s much more to talk about with regards to how Coronavirus exposes our cognitive biases, we’ll dive more into that in another newsletter. In the meantime, I’d recommend reading Cass Sustein’s piece in Bloomberg Opinion.
Sign up here to get notified about my future posts !
What caught my attention this week
“It’s overwhelming”: On the frontline to combat coronavirus ‘fake news’ by Politico. ‘Fake News’ have sadly become mainstream in the past years and seem to worsen every election cycle. However, the global nature of this pandemic has given fake news global proportions. From “the Balkans to Brazil” fact-checkers are struggling to keep up with the pace, putting people’s lives in jeopardy.
Emmanuel Macron’s FT Interview: It’s time to think the unthinkable. All elements are there for a good interview: the context, the politician and the interviewer. This rather short exchange provides a profound and pertinent reflexion on multilateralism, globalisation, Europe and much more.
“Unlike other world leaders, from Donald Trump in the US to Xi Jinping in China, who are trying to return their countries to where they were before the pandemic, the 42-year-old Mr Macron says he sees the crisis as an existential event for humanity that will change the nature of globalisation and the structure of international capitalism.”
Chunky Monkey & Funky Hippies
This week’s podcast recommendation is How I Built This with Guy Raz: Ben & Jerry’s. These past weeks have been quite tough for many of us, so I’m sure you’ll enjoy this good vibes-filled podcast, where Ben & Jerry’s founders, Ben Cohen and Jerry Greenfield, discuss the story behind the creation of their world famous ice-cream. You’ll also learn about failure, business and social impact.
What I’m reading
80 000 Hours: Find a fulfilling career that does good, by Ben Todd
“You have about 80,000 hours in your career: 40 hours a week, 50 weeks a year, for 40 years.
If you could make your career just 1% higher impact, or 1% more enjoyable, it would be worth spending up to 1% of your career doing so. That’s 800 hours – five months of full-time work. We’re going to take little more than a weekend.”
Something to Think About
“There is no reason why because it is dark you should look at things differently from when it is light.” Ernest Hemingway, The Sun Also Rises
Feedback is greatly appreciated! Please reach out on Twitter @the_simonpastor or to simonjpastor@gmail.org !
You should read "Freakonomics". It's basically a whole book's worth of examples of how statistics can be misused and how thinking outside the box (e.g. not just looking at the planes that made it back) can yield surprisingly accurate results.