How can you use statistics to correct intuitions? - Chapter 21

Psychologist Meehl reviewed the results of studies that had assessed whether ‘clinical predictions’ based on the subjective impressions of trained professionals were more accurate than ‘statistical predictions’ made by combining ratings or scores according to a rule. In one study, trained counselors were asked to predict the grades of students at the end of their first school year. They interviewed the students and had access to personal statements, aptitude tests and their high school grades. The statistical formula used only one aptitude test and high school grades, but was more accurate than 11 out of 14 counselors. Other study reviews showed similar results (regarding a variety of predictions: criminal recidivism, parole violations, success in pilot training).

The outcome shocked clinical psychologist and lead to many more studies. But fifty years later, algorithms still score better than humans. 60% of the research shows that algorithms have better accuracy, other studies resulted into a tie. Domains that involve a fair amount of unpredictability and uncertainty are called ‘low-validity environments’. Examples are medical variables (longevity of patients, diagnoses of diseases, length of hospital stay), economic measures (prospects of success, assessments of credit risks) and governmental interests (odds of recidivism, likelihood of criminal behavior). In all these cases, the accuracy of algorithm was better or equally good.

Simple statistics beat the predictions of world-renowned professionals. Meehl’s explanation is that experts try to be smart, consider complex combinations of features and think outside the box. Complexity usually reduces validity. Research has shown that human experts are inferior to formulas even when they are handed the score predicted by the formula. They believe they can do better than the formula because they have more information about the case. Another explanation is that people are inconsistent in making summary judgments of complex information. Two evaluations of the same information result often into two different answers. This inconsistency is probably caused by System 1’s need for context. Unnoticed stimuli in our environment influence our actions and thoughts.

Meehl’s research indicates that final decisions should be made by formulas, particularly in low-validity environments. The final selection of students for medical schools is often determined by interviewing the candidates, which reduces the accuracy of the selection procedure. Interviewers have too much confidence in their intuitions and favor their impressions over other information sources, which reduces validity.

The dominant statistical practice in social sciences is assigning weight to several predictors by following the formula ‘multiple regression’. Robyn Dawes argues that this complex statistical algorithm is rather worthless. Recent studies show that formulas that assign equal weight to all the predictors are best, because they are not affected by sampling accidents. Equal-weighting has a major advantage: useful algorithms can be developed without any previous statistical research, Simple equally weighted formulas based on common sense or on existing statistics are excellent predictors of significant outcomes.

Clinical psychologists received Meehl’s finding with disbelief and hostility, due to the illusion of skill regarding their ability to make long-term predictions. Right judgments are often short-term predictions. The hostility towards formulas will probably diminish, as their value in our daily lives becomes more and more visible. Examples are recommendations by software, decisions about credit limits, health guidelines and the payment of sportsmen.

Imagine you want to hire the best possible person for a job in your company. You should start with selecting a maximum of six independent traits that are required for the position. You must be able to assess the traits reliably by asking some factual questions. Make a list of the questions per trait and think of a scoring scale, for instance 1 (very weak) - 5 (very strong). These preparations take little effort but can make a big difference in the quality of the hired people. Collect information on traits one by one, score the questions before you move on to the next trait, in order to avoid the halo effect. The candidate with the highest score should be hired, even if you like someone else better.