Making Better Forecasts
why it's important and a process for doing it
Forecasting is as much an art as it is a science. It demands a careful balance of quantitative analysis, logical reasoning, and intuitive judgement. In this post, I’ll discuss some of the techniques used by top forecasters and describe how they can be used to make better forecasts.
Why Forecast?
Why is it important to forecast? Imagine steering a ship through fog. You can't see far ahead - it might be open water, but it could also be jagged rocks. By forecasting, we send out 'radar pings', giving us information about potential obstacles or safe passages. These signals help us identify risks and benefits, enabling us to move away from less desirable futures and toward better ones.
Ask Better Questions
It’s important to follow a structured approach to forecasting that begins with clear, unambiguous, and falsifiable questions. Instead of asking 'Is humanity on the verge of catastrophe?' ask 'Before 2070, will the human population be reduced to below 4 billion?' The latter question sets a specific timeline (before 2070) and a quantifiable metric (human population reduced to below 4 billion). This is clear, unambiguous, and falsifiable, meaning it is a question whose answer can be confirmed or refuted based on future data.
Enumerate Outcomes
By identifying and exploring potential futures, we can better assess their probabilities and implications, guiding our actions toward risk mitigation and opportunity capitalization. Since the total outcome space must sum to 100%, seeing all the potential outcomes together serves as a coherence check (e.g. it will prevent mistakes like we sometimes see in prediction markets where the outcomes sum to more than 100% as shown below). Also, unless you are very confident of the outcome space, it’s a good idea to save room for unexpected / unmodeled outcomes.
Anchor on Base Rates
In many circumstances, particularly those where we have relevant historical data, it is beneficial to employ an 'outside view'. This involves anchoring our initial estimates on base rates derived from similar historical contexts. After establishing this starting point, we then make adjustments based on specific attributes or components of the current scenario. This approach, as outlined in Superforecasting by Tetlock and Gardner, has been demonstrated to significantly improve forecast accuracy in numerous empirical studies. However, it's critical to remember that while base rates offer a useful starting point, they are not definitive. We must still be prepared to revise our forecasts in light of new information.
Break Questions Down
For some factors, a ladder-style decomposition where every step is conditional on the previous step is appropriate. For example, in predicting whether a fair coin will land on heads twice in a row, the first flip must land on heads and then the second, so this is easy to decompose as (0.5) * (0.5).
While ladder-style decomposition is beneficial for understanding linear, conditional dependencies, many questions involve interrelated factors. In such cases, a belief network approach is more suitable. Consider how, in the context of the United States and China, an AI treaty might impact the probability of an AI arms race or vice versa. The presence of one may reduce the chance of the other, but it is possible for both to occur in the same universe.
Identify Risk and Uncertainty
Risk pertains to situations where we have enough historical data to estimate the probabilities of different outcomes. Here, the challenge lies in accurately interpreting and applying this data to future scenarios.
Uncertainty, on the other hand, arises in situations where we lack enough data to estimate probabilities. In these instances, forecasting becomes more about exploring a wide range of possible outcomes, identifying key influencing factors, and using logical reasoning to evaluate their potential effects.
Risk-based questions typically have lots of relevant data and can be calculated with a relatively high degree of confidence ( e.g. will a fair six-sided die land on 4?).
Uncertainty-based questions aren’t as easily quantifiable (e.g. By 2035, will there be a widely accepted theoretical framework that allows for the confirmation of qualia or subjective experiences within advanced AI systems as confirmed by at least 70% agreement in a survey of AI researchers?).
Most questions fall somewhere in between, so try to squeeze out the maximum amount of risk calculation from any question.
Empiricism Over Theory
"It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong." - Richard Feynman
It can be tempting to rely heavily on theoretical models. Theories, with their elegant mathematical formulations and seductive intellectual purity, offer a seemingly neat and logical way to predict the future. However, they are, by their very nature, simplifications of the real world. They are built on assumptions, which may not hold true in every context or situation. In many cases, they may fail to capture the full complexity and nuance of reality.
This is why we emphasize the primacy of empiricism over theory in our forecasting approach. Empirical evidence, gathered from careful observation and rigorous experimentation, gives us direct insights into the world as it is, not as our theories suggest it should be.
That's not to say that theory has no place in forecasting. Indeed, theoretical models play a crucial role in guiding our inquiries and helping us make sense of the evidence we gather. But they should be treated as tools to aid our understanding, not infallible oracles dictating what the future must be.
When empirical evidence contradicts our theoretical predictions, we should be willing to adjust or even discard our theories. We need to remember that our ultimate aim is not to vindicate our pet theories but to accurately forecast potential outcomes.
Use Logic
In situations where empirical data is sparse or non-existent, logical reasoning becomes a critical tool. This can involve techniques such as creating theoretical models, establishing upper and lower bounds based on logical limits, and drawing inferences from available knowledge. The use of logic in forecasting is not a fallback for when data is unavailable; it is an integral component of the process that works hand-in-hand with empirical analysis. Both data-driven and logic-based reasoning must be used in concert to navigate the complexity of forecasting outcomes, and indeed, any complex future scenario. Remember: data illuminates the path, but logic guides us through the dark corners where data is yet to shine.
Perform Premortems
Start from the hypothetical premise that your forecast was wrong and work backward from the outcome to determine what happened. Consider the potential off-ramps from the path you considered most likely and think about what evidence would change your mind about your conclusions. Identify the most likely issues and re-evaluate your forecast in light of this information.
Place Your Bet
Assess your confidence. How much would you be willing to bet on your forecast and what odds?
Consider your track record. Do you have a history of overconfidence? If so, be humble and bias toward uncertainty.
Do people or sources you respect disagree? If so, it might be worth moving in their direction a little, particularly if they have good empirical track records.
Finally mark down your forecast. You’ve done the best you can. If you’re wrong, you’ll learn from the experience and improve.
Make Updates
Forecasts shouldn’t be treated as ideological convictions to be defended. As new evidence emerges, be flexible and make updates to your beliefs, and always be on the lookout for disconfirming evidence as you established during your premortem.
Conclusion
Following these suggestions won’t give you a crystal ball, but they might help you avoid some of the worst mistakes. As with anything practice makes perfect and as you get more forecasts under your belt this will give you data that will help you improve your calibration.