Art to Science

forecasting bots will transform the field

Since its beginning, forecasting has been more craft than science. There were theories about what works (e.g. base rates) but testing them was slow, expensive, and small-scale. Forecasters could disagree about method and never get to a resolution because the feedback loops were long and sample sizes were too small.

That’s changing.

Forecasting bots are automated systems that take questions - “Will X happen by Y date?” - and output probability estimates. They can run 24/7, answer hundreds of questions, and get scored against outcomes. The Metaculus AI Benchmark is now running tournaments with hundreds of questions and real-time feedback. We can finally test forecasting methodologies empirically at scale.

This is forecasting’s ImageNet moment. ImageNet was the dataset that transformed computer vision from an academic backwater into the foundation of modern AI. Before ImageNet, researchers argued about which approaches to image recognition were best. After ImageNet, they could just run experiments and see.

The same thing is about to happen to forecasting. We’re going to learn what actually works.

I’ve written before about simple forecasting instructions that get you 80% of the way there, but building a bot that performs at the frontier requires going deeper. The simple heuristics are table stakes.

I’m building a bot. And I’m going to reveal my methodology in a series of posts. After the tournament ends, I’ll release everything.

The field is about to take a leap forward. Might as well document it in real time.