In Superforecasting Phil Tetlock explores how his team of volunteer amateurs beat CIA analysts and University researchers at forecasting complex events.
I draw out three points:
1- Some context on the project and forecasting in general. How do you measure a forecaster’s accuracy? What have a talking-head forecaster and a dart-wielding chimp got in common?
2- How to make a good forecast: Break the question down. Start from the right base-rate (a comparable statistic). Add the details and consider other perspectives.
3- How can you become a better forecaster? Chiefly, practice, get honest feedback, build a broad knowledge base, and update yourself until you trust yourself.
Want to know more?
This blog post is part of a series I am making called Reading For The Aspirational Self. Don’t think of this as book summaries – I’m not doing that. Instead, I’m drawing out specific lessons that I find particularly interesting. And which I think could act, together, to help people who share my aspirations. If you, too, want to be present, family-centric, intrinsically motivated and polymathic, I can help.
- The most distilled version of what I’m offering is a free mailing list designed for learning, “Think On Thursday” – each e-mail will include a lesson designed around the content. Click here for some information on that.
- The series is also on YouTube in the form of 7-12 minute videos. Here’s the channel link – the video and transcript are below.
- I’m tweeting excerpts from the videos, as well as some of the story of this project, how we’re doing it, and where it is going, on Twitter. @DaveCBeck
If you want to know more about Reader, Come Home take a look at The Good Judgement Project’s website here.
This week’s video:-
Forecasting complex events is a really difficult thing to do, but some people are measurably better at it. In Superforecasting Philip Tetlock investigates what it was that allowed his teams of volunteer forecasters to beat university researchers and CIA analysts over a five-year period.
In this video I’ll firstly explore the Superforecasting project and talk a little bit about forecasting in general. Then I’ll draw out two key lessons. Firstly, how can you make a good forecast if you’re just making one? Secondly, what attributes make a better forecaster? Most of these are things that you can work on if forecasting the future matters to you.
Forecasting complex real world events is a tough thing to do well to see how difficult, just look at talking head forecasters the people you see on the TV and in the media and read about on, on the news and you see them forecasting elections, and you see them forecasting where the markets are going next with complete certainty because certainty sells.
But when you actually examine their records, they’re terrible. The, the joke that. Tetlock made was that they were about as accurate as dart throwing chimpanzees. So if talking head forecasters to people who are paid to do it, can’t forecast particularly well, who can? That’s one of the questions that underpin something called the good judgment project, which this book was based upon.
(section – Forecasting, and the Good Judgement Project)
The good judgment project has been running since 2011. And over 20,000 volunteer forecasters have been involved. These are all people who either work or retired, from a variety of backgrounds and they contribute to the project. They contribute their best estimates as to what will happen next on things like world events, things like the economy.
Recently there have been quite a few questions on technology as well and how that will develop in the coming years. And what they’re doing is together they’re forecasting. And the research questions behind it are what makes some people better forecasters than others? That’s the question that Tetlock was examining through this book and through the project as a whole.
The project was a huge success and the good judgment project, these volunteer forecasters, when they all group together, they beat university research teams at forecasting world events. They beat CIA analysts, all of whom had access to classified information, which these volunteer researchers didn’t have.
They were more accurate over a five-year period of forecasting world events, the economy, and much else besides. Working in teams, they also beat the prediction markets, which are seen as quite an accurate representation because people put money behind their bets on prediction markets. And there is the what’s called the wisdom of crowns, a harness there too, but the superforecasters working in teams were more accurate than those.
(section – How to make a forecast)
The type of things they were forecasting are what Tetlock called Goldilocks questions. So they’re not too hard, they’re not impossible to forecast or completely subject to chance, but they’re not too easy either. So you can’t make a very simple, quick prediction about what’s going to happen next. I’ll throw a couple of examples up here and just to show you the kind of questions, I mean, straight from the good judgment project website, so that they’re not too hard, they’re not too difficult.
And importantly, they’re measurable. They have an outcome that at the end of it, when you look at what happened, you can tell whether the forecaster was right or wrong, just by looking at it in a way that is not particularly biased and is clear. So the forecast that they’re making have to be measurable as well.
They can’t be saying, Oh, I think this might happen. They have to give a number to that might. And that kind of numerical precision is very important. If you’re going to work on your own forecasting, or if you want to assess how much credence to give to other people’s forecasts as well.
And the way that they measure the success of somebody forecasting is through something called the Brier score. So the Brier scores are a little bit like a pass score in golf. The closer you are to 0, the better you’re doing. It ranges up to 2 for people who get everything wrong and then lower and lower and lower as you get towards 0.
(Section – Making a good forecast)
There are three key elements to a good measurable forecast. The first is to assign a percentage probability. Don’t use words. One of the most interesting things, actually the most amusing things from Tetlock’s work was a comparison of what people considered a percentage probability attached to a word wall.
So, when somebody says might, what do they mean by that? And people’s estimates vary hugely. There is no way to compare different people’s forecasts of something that might or almost certainly, or probably will happen. Whereas once you give a percentage probability, what you’re, what you’re doing is giving a measurable outcome, probability based outcome.
The second thing is I mentioned is a determinable event, some things that either will or won’t happen. And the third thing is a timeframe. By giving a timeframe. You’re both encouraging yourself to look at the appropriate levels of complexity and to think about different things that can happen over the span of six months, as opposed to six years.
And secondly, it’s about the measurability. So it helps ensure that an event can or does not happen by that certain date.
**I think the section break should be here.
If you’re trying to predict something from the future. Firstly, slow down. Try and suppress your intuitive first response when dealing with complexity, because most of you haven’t trained your intuition to deal with that yet. And use system two thinking, use your slower causal reasoning processes to work out how important the different factors are.
Secondly, answer the question that you’re being asked, not an easier one. A lot of people substitute what do the authorities think on this question? And when dealing with things like the markets or dealing with things like political events, we often don’t have access to people who are actually knowledgeable about those things.
We have access to people who are loud and talk a lot, and while they might sound knowledgeable because of their certainty, their opinions are as accurate as dart, throwing chimpanzees that they’re worthless when it comes to predicting the future. So, if you’re trying to make a prediction yourself, you need to try and tune that noise out and answer the question that’s asked.
Thirdly, break down the question into its component segments. So try and pull a question apart and work out what the different factors that have probabilities of occurring are that make up the overall question. This is something called the Fermi method after and Enrico Fermi. And what it does is it encourages you to think about the different parts. The different factors that influence a question.
The fourth kind of key is to do things in the right order. So rather than trying to come up with an overall solution at once, try and find a base rate either for all of the component parts or for the overall outcome, look for something that gives you an idea of how likely that thing is to have happened in that timeframe, over the past historical record.
For most things that you’re trying to predict, there is a comparable base rate that you can use. And if you start from that base rate, what it does is it lets your anchoring bias work for you.
Once you’ve got that base rate and you, you, you know where to start from, you start to think about the more detailed issues. So you look into why is it different this time?
What’s unusual about the present case as opposed to the historical record. And how does that affect the probability of the event that I’m looking at happening? So, as you build more of that detail and more of that complexity, you’re moving your probability estimate from 60 to 63 or to 67% or wherever it goes, moving it up and down the scale in the face of the new information you come across.
And as you add more detail, you’ll feel more comfortable with this as well. And the key thing to do towards the end. And this is most useful towards the end. You have a personal, personal guesstimate, personal forecast is to consider other perspectives, try and hold some alternative narratives in your head and allow yourself to, to kind of listen to other people’s opinions at this point and weigh them, compare them with your own and weigh them. Maybe somebody else has seen something you haven’t. Particularly. Again, if you’ve broken the question down into component parts, you might find that one of your component parts was way off.
One of the other things that the superforecasters did better than nearly anybody else was they dealt with timeframe better. As a general rule. The longer the timeframe, the more likely something is to happen. Just because if it could happen in the short amount of time, it could also happen in the longer amount of time.
(section – how to become a better forecaster)
Firstly, if you’re working in a team. Divide the cognitive labor up, maybe split the question into different component parts and assign each one to somebody else and try and tread the line between aggressive, dysfunctional team and Groupthink. The easiest way to do this is to only ask precise questions. Don’t ask very general questions, try and narrow in on details because once you’re discussing details, conversations tend to be more productive.
The analogy that Tetlock used was from Isaiah Berlin. He said, be a Fox, not a hedgehog. So what he means by this is hedgehogs are very specialist creatures. They can look with great detail at what they do, but they don’t know anything about the wider world. Whereas Foxes is roam across different domains. So if you want to be about a forecaster. The lesson to draw is that you need to be more Fox-like. You need to draw your knowledge from different domains, know a little bit about things that aren’t your area of expertise, just so that you can take them into account, particularly in predicting outliers and unusual events. It helps if you understand how other people see the world, as well as how you understand it yourself.
And the other analogy that Tetlock used is dragonfly eye. So the ability to consider multiple perspectives is really important in making a good forecast. And this is something he consistently raised, the idea of multiple perspectives.
So, the superforecasters ruled relatively intelligent, probably in the top 20% by IQ, but not at genius level, whether that’s because there were no geniuses participating or whether that’s because, for some reason, those of, of a greater intelligence can’t apply that intelligence to forecasting is a different question.
But most of the things that made the superforecasters stand out as compared to the rest of the forecast teams who didn’t do so well as part of the good judgment project were mindset based.
The chief one is a willingness to grow and update. It’s a mindset that Tetlock referred to as perpetual beta. The idea that you’re continually developing both your knowledge on the forecasts and you don’t get hung up on the forecast you made six months ago.
If there’s some new information that comes out, that suggests you should change it quite significantly, you just change it. There isn’t that emotional attachment to the forecast that you’ve made into the person you were six months ago, the estimate you made back then. And that ability to change and progress as an individual was a key distinguishing factor.
The superforecasters were able to filter out irrelevant information really well. They had a process in place, or they just had the mindset where if something wasn’t directly related to the question at hand, they didn’t allow it to affect them.
There were numerates, they understood how probability worked, and this is something that you can learn. You don’t need to get into the details of Bayesian theorem. You just need to know basic percentage probabilities and how to estimate those and what they mean in the real world. Any of you who’ve done any form of gambling are probably numerate enough to make a decent forecaster.
You just need to work on the precision of estimates that you’re making. That’s really the key. It’s, it’s turning a basic numeracy into measurable predictions.
You need to learn from success as well as from failure. There’s a widespread belief that if we fail, we need to learn and then iterate and do something better again, next time.
But if we succeed. We might have only succeeded because a couple of variables fell into place for us. There is nearly always something to learn from real events when you’re making complex forecasts that you got wrong. Some sub-component of a prediction and being able to learn from success as well as failure was another thing that stood out about the way that the superforecasters discussed their practice among forecasting teams.
The key thing to become a better forecaster is to practice, to make measurable predictions. To learn from them, whether you were right or wrong and to iterate and get better as you go along. And what practice does is it, doesn’t just update your system to thinking you’re slow thinking and allow you to take account of things you didn’t previously take into account.
It also begins to train your intuition. So to give you an example of that. Firefighters who are deciding whether to go into a burning building or not, they don’t build a probability model and try and guess the chance of it falling down on their heads on a percentage basis. They listen, they listen to their intuition.
If they feel uncomfortable, they don’t go in. If they feel like it’s a risk worth taking, they do. And they make these intuitive judgments under immense pressure and they get it right. Phenomenally, commonly, and they don’t know why. So when they’re having their after-action debriefs, that they don’t know why they made those decisions, they can rationalize it post hoc, but it’s their intuition that they’re listening to.
And it’s a trained intuition. The same thing happened with the superforecasters. They generally assuming external factors in their lives didn’t intervene because these were volunteers, remember. They generally got better over time, better over the first few years.
And that’s partly pattern recognition. It’s partly self-belief. And it’s partly this trained intuition that I’m talking about too.