Tuesday, May 28, 2013

"Survivorship Bias"

That's the title of a fascinating article by David McRaney at his Web site 'You Are Not So Smart'.  He discusses statistical analysis in World War II, shows how a very bright statistician prevented the US Army Air Force from making a disastrous mistake, and demonstrates how that illustrates a problem in decision-making that affects us all.

Simply put, survivorship bias is your tendency to focus on survivors instead of whatever you would call a non-survivor depending on the situation. Sometimes that means you tend to focus on the living instead of the dead, or on winners instead of losers, or on successes instead of failures. In Wald’s problem, the military focused on the planes that made it home and almost made a terrible decision because they ignored the ones that got shot down.

It is easy to do. After any process that leaves behind survivors, the non-survivors are often destroyed or muted or removed from your view. If failures becomes invisible, then naturally you will pay more attention to successes. Not only do you fail to recognize that what is missing might have held important information, you fail to recognize that there is missing information at all.

You must remind yourself that when you start to pick apart winners and losers, successes and failures, the living and dead, that by paying attention to one side of that equation you are always neglecting the other. If you are thinking about opening a restaurant because there are so many successful restaurants in your hometown, you are ignoring the fact the only successful restaurants survive to become examples. Maybe on average 90 percent of restaurants in your city fail in the first year. You can’t see all those failures because when they fail they also disappear from view. As Nassim Taleb writes in his book The Black Swan, “The cemetery of failed restaurants is very silent.” Of course the few that don’t fail in that deadly of an environment are wildly successful because only the very best and the very lucky can survive. All you are left with are super successes, and looking at them day after day you might think it’s a great business to get into when you are actually seeing evidence that you should avoid it.

. . .

Wiseman speculated that what we call luck is actually a pattern of behaviors that coincide with a style of understanding and interacting with the events and people you encounter throughout life. Unlucky people are narrowly focused, he observed. They crave security and tend to be more anxious, and instead of wading into the sea of random chance open to what may come, they remain fixated on controlling the situation, on seeking a specific goal. As a result, they miss out on the thousands of opportunities that may float by. Lucky people tend to constantly change routines and seek out new experiences. Wiseman saw that the people who considered themselves lucky, and who then did actually demonstrate luck was on their side over the course of a decade, tended to place themselves into situations where anything could happen more often and thus exposed themselves to more random chance than did unlucky people. The lucky try more things, and fail more often, but when they fail they shrug it off and try something else. Occasionally, things work out.

. . .

You succumb to survivorship bias because you are innately terrible with statistics. For instance, if you seek advice from a very old person about how to become very old, the only person who can provide you an answer is a person who is not dead. The people who made the poor health choices you should avoid are now resting in the earth and can’t tell you about those bad choices anymore. That’s why it’s difficult not to furrow your brow and wonder why you keep paying for a gym membership when Willard Scott showcases the birthday of a 110-year-old woman who claims the source of her longevity is a daily regimen of cigarillos, cheese sticks, and Wild Turkey cut with maple syrup and Robitussin. You miss that people like her represent a very small number of the living. They are on the thin end of a bell curve. There is a much larger pool of people who basically drank bacon grease for breakfast and didn’t live long enough to appear on television. Most people can’t chug bourbon and gravy for a lifetime and expect to become an octogenarian, but the unusually lucky handful who can tend to stand out precisely because they are alive and talking.

. . .

Failure to look for what is missing is a common shortcoming, not just within yourself but also within the institutions that surround you. A commenter at an Internet watering hole for introverts called the INTJForum explained it with this example: when a company performs a survey about job satisfaction the only people who can fill out that survey are people who still work at the company. Everyone who might have quit out of dissatisfaction is no longer around to explain why. Such data mining fails to capture the only thing it is designed to measure, but unless management is aware of survivorship bias things will continue to seem peachy on paper. In finance, this is a common pitfall. The economist Mark Klinedinst explained to me that mutual funds, companies that offer stock portfolios, routinely prune out underperforming investments. “When a mutual fund tells you, ‘The last five years we had 10 percent on average return,’ well, the companies that didn’t have high returns folded or were taken over by companies that were more lucky.” The health of the companies they offer isn’t an indication of the mutual fund’s skill at picking stocks, said Klinedinst, because they’ve deleted failures from their offerings. All you ever see are the successes. That’s true for many, many elements of life. Money experts who made great guesses in the past are considered soothsayers because their counterparts who made equally risky moves that failed nosedived into obscurity and are now no longer playing the game. Whole nations left standing after wars and economic struggles pump fists of nationalism assuming that their good outcomes resulted from wise decisions, but they can never know for sure.

There's much more at the link.  Go read the whole thing - it'll challenge all sorts of preconceived ideas.

This is a truly fascinating article.  I can't recommend it too highly to anyone who has important life decisions to make.  I'm certainly going to apply it to decisions I have to make about my (hopefully burgeoning) career as an author.



STxRynn said...

Okay, I'm late for work and it's your fault!! This was an excellent article. I realized that my work experience has been filled with survivorship bias.

When I came to know Jesus as my saviour, I was encouraged to take every ministry opportunity that presented itself. No coincidences so to speak. I've taught sunday school from first graders to senior citizens , pastored a church, mopped floors, helped the homeless, supported missionaries and all the while not realizing I was looking for everything that was there.

At work, I'm very conservative and loyal, I won't look for other opportunities unless it's obvious the ship is about to founder.

This was SO timely in my situation, and I want to THANK YOU from the bottom of my heart for posting it.

The application of this concept is staggering my imagination....

Peripatetic Engineer said...

Engineers knkow that the study of their failures is essential if you want to improve your design. I tutor my 8 year old grandson in math. I tell him that we have to look at the problems he got wrong and study those problems. The other stuff he got right and doesn't need to study it as hard. I tell him that good scientists do the same thing. I'm sure it was painful for him to review a paper with a bad grade but it helped us to figure out what he was doing wrong and fix it.

Chris said...

I knew this as a vague axiom, that you learn from failure, not success. However, I have not, up to now, REALLY thought about it. I am going to circulate this article among my co-workers.

Retired Mustang said...

One of my mentors (a truly overused term, I know) said two things that apply here. First, when it comes to finances, he said I should learn what poor people do and do something else. The second was that it's not enough to learn from the successful in any area of life. He said it would be great if the unsuccessful gave seminars...or if each of us could find a man who has made a wreck of his life for 50 years and follow him around for a while and take notes.