The world is facing serious threats, but we can't even agree which ones are real. Deciding which to take seriously without testing our models is like betting your life savings on a weather forecast based on a groundhog's shadow.
World Models Described
Everyone has a world model, whether they realize it or not. It's an internal representation of reality – the beliefs, assumptions, and mental shortcuts used to make sense of a complex world.
When you wake up in the morning and plan your day, you're using your world model. You assume the sun will rise, your lights will come on when you flip the switch, and the world outside your house will exist when you go out the door.
The problem is, our world models can be flawed, particularly when we don’t get direct feedback on them. They can be based on incomplete information, biased by our personal experiences, or outdated. It’s not a big deal when the stakes are low and we find out that while we had no trouble explaining how an airplane can fly normally, we can’t explain how it can fly upside-down. Unfortunately when we're dealing with global challenges, the limitations of our world models become important.
Evaluating Competing World Models
The first challenge is that there are a lot of competing world models to consider, so many in fact that evaluating them all would be an intractable problem. In virtually any given field, even if there are many points of consensus, you'll still find areas where there is wild disagreement.
On top of that, we face the problem that every now and again the prevailing paradigm is flawed, but gathering the data to overturn it could take a long time to collect. Existing paradigms often become entrenched not just due to scientific inertia, but also because of ego investments, tribal affiliations within academic or professional communities, and power structures that benefit from the status quo.
Even if a model has good explanatory power, it might still be spurious. Consider the classic example of the correlation between ice cream sales and drowning incidents. A model based on this correlation might appear to have strong predictive power, but it misses the true causal factor (warm weather) that influences both variables.
Evaluating Models with Instrumentalism
It probably goes without saying that some models produce better results than others. Consider this passage from Sapiens.
In 1744, two Presbyterian clergymen in Scotland, Alexander Webster and Robert Wallace, decided to set up a life-insurance fund that would provide pensions for the widows and orphans of dead clergymen. They proposed that each of their church’s ministers would pay a small portion of his income into the fund, which would invest the money. If a minister died, his widow would receive dividends on the fund’s profits. This would allow her to live comfortably for the rest of her life. But to determine how much the ministers had to pay in so that the fund would have enough money to live up to its obligations, Webster and Wallace had to be able to predict how many ministers would die each year, how many widows and orphans they would leave behind, and by how many years the widows would outlive their husbands.
Take note of what the two churchmen did not do. They did not pray to God to reveal the answer. Nor did they search for an answer in the Holy Scriptures or among the works of ancient theologians. Nor did they enter into an abstract philosophical disputation. Being Scots, they were practical types. So they contacted a professor of mathematics from the University of Edinburgh, Colin Maclaurin. The three of them collected data on the ages at which people died and used these to calculate how many ministers were likely to pass away in any given year.
How do we differentiate between competing world models? It depends on the goal, of course. For some, it's about forming a consistent paradigm, for others it's a grab bag of ad hoc rules that work well enough to get by without thinking too hard, and for others it may be more about finding a set of beliefs that allows for minimum conflict within their tribe. For most readers here, however, I suspect it is more about practical utility: using models that help us accurately predict and navigate reality.
This pragmatic approach, known as instrumentalism, judges models not by whether they're "true" in some abstract sense, but by whether they're useful tools for understanding and interacting with the world. Under instrumentalism, the best model is the one that most reliably generates accurate predictions and enables effective intervention.
Consider how we evaluate scientific theories: Newton's laws of motion aren't considered valuable because they represent some perfect, ultimate truth, in fact, we know they break down at quantum scales. They're valuable because they allow us to accurately predict how objects will behave under everyday conditions.
This is where good forecasters enter the picture. These individuals and teams have empirical track records of predicting future events across domains. While we'll explore the cognitive mechanisms behind their success in a future article, the important point is that their methods work. By identifying who consistently makes accurate predictions, we gain a shortcut to identifying which world models are most effective for understanding reality.
The Cheat Code
One might ask: How do we know forecasters aren't just identifying spurious correlations themselves? After all, we've seen that correlations can appear predictive without revealing true causation.
What sets good forecasters apart is their consistent performance across domains and time horizons. Unlike the ice cream-drowning correlation that breaks down when conditions change, skilled forecasters maintain accuracy through shifting circumstances.
Research shows that forecasters with good track records tend to maintain their high performance across various domains, making well-calibrated predictions even outside their areas of expertise.
Groups of forecasters consistently outperform individuals. These collaborative efforts combine diverse perspectives and analytical approaches to produce accurate predictions, often surpassing traditional domain experts.
Consider this passage from my fellow forecaster Molly Hickman’s recent article that she wrote in conjunction with Rajashree Agrawal:
The foundational literature (e.g. Goldstein et al, Karvetski et al, Tetlock's Expert Political Judgment) in Tetlockian forecasting seems to point at the top forecasters having better understanding of how the world works than domain experts (if you're on board with the idea that proper scores, in this case Brier, are a reasonable proxy for "understanding of how the world works"). Despite lacking the years of knowledge, experience, and gut-feelings that experts have, forecasters are better at putting numbers to whatever knowledge, experience, and gut-feelings they do have—implying that high-quality expression of shallow information > low-quality expression of deep information. This makes some intuitive sense; for instance, despite farmers' having much deeper understanding of agriculture, it doesn't seem like they're better at crop futures trading than traders.
These forecasters have the right incentives. They stake their reputations on their predictions, aligning their interests with the pursuit of truth rather than supporting particular agendas or maintaining the status quo.
Rather than endless theoretical debates or waiting years for definitive evidence, we can look to these forecasting groups as our reality-checking mechanism and a practical means of determining which world models deserve our attention and resources.