Suppose you are bitten by a rattlesnake. Would you rather be treated by a medical toxicologist, or by a faith healer who chews up herbs and spits them into the wound?

Because snake bites usually aren’t fatal, most patients survive either way. In noisy domains, almost anyone can claim some surviving patients. The question is not whether you can survive a bad process. The question is which model actually improves your odds.

In public reasoning, we choose the faith healer constantly. Most people form beliefs using heuristics like narrative fit or social identity that were good enough for natural selection. This might work for navigating social environments, but it isn’t suited for developing accurate world models. Without external feedback loops, it’s too easy to reinterpret contrary evidence and explain away failures. The deeper problem is that most people have no reliable mechanism for even noticing when they’re mistaken.

The Exam

A story can be crafted to explain anything after the fact, and a theory that doesn’t make falsifiable predictions can survive indefinitely. Forecasting brings the right incentives for beliefs to move from trying to sound good to trying to be predictive.

When I’m teaching, I want my students to do well, but I also want them to comprehend the material. I can’t just give them the exam key to study from or most of them would just overfit to the test. The purpose is to find out if they understand the material well enough to solve new problems.

World models should be judged the same way.

A model that only explains the past may be nothing more than a story fitted to the answer key. A model that anticipates new cases has captured something real. Forecasting demonstrates whether a world model holds up outside the training set.

Forecasting is an exam for world models. A strong track record is compressed evidence about whose beliefs are disciplined by reality. Top forecasters have demonstrated that their expectations are aligned with how the world actually is.

Top Performers

If forecasting is the exam reality gives to world models, forecasters who consistently outperform the crowd are the students at the top of the class.

Marginal advantages matter. An investor does not need to beat the market every day to have skill. Similarly, a good forecast usually isn’t “the crowd is at 20%, but I alone know it’s 100%.” More often, the advantage looks like slightly better calibration over time.

Forecast scoring separates people with reality-shaped models from the rest. These people have demonstrated one of the rarest and most valuable intellectual skills: the ability to form beliefs that reality vindicates.

The Leaderboard

Building a meritocratic leaderboard was the promise of forecasting platforms.

They would identify people with unusually good judgment, measure their performance over time, and make that talent useful. The idea was to discover people whose beliefs repeatedly made contact with reality, then route them toward decisions where reality-contact mattered.

On the first part, forecasting platforms largely succeeded.

They showed that probabilistic judgment is not just noise. Some people consistently outperform others. They are better calibrated, better at using base rates, better at updating, and less likely to let a preferred story dominate their expectations. Across many questions, these small advantages compound into a meaningful edge.

The Opportunity

The ideas from the forecasting community have started to take hold, but the market for truth is still smaller than the market for reassurance, and many of the most important concepts haven’t been widely adopted yet.

In many cases, the faith healer model limps along simply because of inertia. When you combine entrenched institutional advantages with the natural appeal of a good narrative and drop it all into a noisy world, the faith healer can survive a very long time before anyone starts questioning their track record.

A compelling story will always be easier to absorb than a calibrated probability. But that blind spot is exactly what makes the opportunity so valuable.

Forecasting platforms have identified people whose beliefs have been repeatedly tested against reality. The next step is to route that judgment toward decisions where being wrong is expensive, giving organizations access to something public reasoning usually lacks: a measured record of who is more likely to be right.

About two weeks ago, my friend Varun threw a farewell party. He’s moving to the Bay Area. The apartment was full of researchers, founders, and the kinds of people who get it when it comes to how much AI is about to turn everything on its head, the kind of people where every conversation makes you think I want to talk to this person again.

I left the party with a familiar feeling. I’d just spent an evening with exactly the kind of people I want more of in my life, and I knew that, in all likelihood, I would never see most of them again.

We exchanged a few numbers, sure, but I’ve done this enough times that I know follow ups are rare and when contemplating the cognitive overhead of coordinating two people’s scheduling and location preferences, I usually just file it under the “I’ll think about this later when I have bandwidth” which usually never happens.

This time though, I wanted to try something different. This felt like a software-shaped problem, so, a few days later, I started building.

Epistemic Density

There’s a reason I live in NYC and not the Bay Area. I like having at least some friends with interests outside of technology. But I also sometimes want a particular kind of conversation. I have Claude, ChatGPT, and Gemini subscriptions and actually know what the latest models are capable of and how fast they’re improving. I’m building forecasting bots, experimenting with agents, and working on research. I spend my days thinking about how the world is about to change, and I want to talk about it with people who are thinking about it too.

People talk about the Bay Area as the place to be for this, and there’s something to that, but I think it’s less about geography and more about what I’d call epistemic density: the odds that the person next to you at a coffee shop has updated on what AI means for everything.

In New York, I’m surrounded by brilliant people, but most of them don’t spend their weekends building agents. That’s fine. They have rich lives and I appreciate the conversations I have with them, but I also want conversations where I don’t have to start from scratch explaining why this matters.

My friend Ben Turtel, who runs an AI startup here in NYC, put it perfectly: his friends treat the things he says about current reality as predictions… that are wrong! That’s my experience too. I want to talk to more people who “get it”, but, right now, it feels like they’re scattered around the city and I don’t know how to find them. Maybe they’re not so scattered though. Maybe they’re just around the corner and we just haven’t found each other yet.

Closer Than You Think

Steve Jobs said “If you are willing to travel around the world to meet a teacher, one will appear next door”.

The people I want to know are often hidden in plain sight. Varun lived five minutes away, yet we were strangers until we met at a Dwarkesh Patel party. Another friend lives even closer, we’d known each other for years, but we only started getting coffee in the last few months after discovering we lived in the same neighborhood at a conference on the other side of town. I have multiple other NYC friends who I only really got to know from events in the Bay Area.

The irony is painful: I travel to niche events to meet people living on my block.

Reducing Friction

The concept was like Calendly, but for coffee meetings. I started working on coffeemeet.xyz the next day, made it live after a few days of testing, and by the following weekend, I was already meeting someone who booked time with me.

The concept is simple: I pick my favorite coffee spots, I set when I’m free, and anyone can book a time to meet me. No back-and-forth texting. No “where works for you?” Just pick a spot, a time, and show up.

For my profile, I picked coffee shops nearby. This isn’t because my time is more important than anyone else’s. It’s because between work, teaching, forecasting, and life with my partner I don’t have many free time slots. If meeting someone new requires an hour-long commute, I know from experience that “let’s do this again” turns into “maybe next month” turns into nothing. The friendships that actually develop are the ones where getting together is easy. Varun was five minutes away and we still only managed coffee once or twice a month. Proximity is what makes consistency possible, and consistency turns conversations into friendship.

Software-Shaped Problems

Here's the thing: the gap between an idea and a working tool has collapsed. Friday night, it was just a feeling at a party. A few days later, it was an app people were using.

I could have built coffeemeet without AI, but “could have” and “would have” are different things. AI didn’t spit out a completed app from a single prompt for me, but it compressed a project that might have taken a couple of weeks into a few hours. And that speed changes the calculus on what’s worth building. Problems that used to feel too small or too personal are now worth a Sunday morning.

I think a lot of people are sitting on little problems like this, software-shaped holes that could be solved if they knew where to start. Not every idea needs to be a startup. Some just need a clear idea, a few hours to work out the kinks, and a willingness to ship something imperfect.

Come Have Coffee With Me

If you’ve read this far, I want to meet you. Seriously.

That’s the whole reason I built this and the whole reason I’m writing about it.

If you’re in or around Brooklyn, or willing to make the trip, pick a spot and a time on coffeemeet.xyz. I’d love to talk about forecasting, AI, what you’re working on, what you’re reading, or whatever’s on your mind.

Some of my best friendships started with a coffee. Maybe ours will too.

When I worked in government, an experienced colleague told me something that’s stayed with me: “Ever since ancient Rome, politicians who build roads win elections.” It’s an achievement people can see, and impossible for opponents to deny.

Housing reform is the centerpiece of the Abundance agenda, and the theory makes sense: America’s cost-of-living crisis stems from a shortage of housing where people need it, and the solution is to build more.

The problem is, if reforms work and rents only rise 2% instead of 6%, no one will notice. Worse, by the time housing construction translates into rent relief (probably 5 years later), someone with a different agenda will be in power. Either way, “Rents didn’t increase as much as they would have” isn’t a ribbon-cutting moment.

The Visibility Problem

Consider Obama’s stimulus tax cuts.

According to Tax Policy Center estimates, 96.9 percent of households enjoyed a tax cut that averaged almost $1,200. Just one measure—Obama’s Making Work Pay tax credit—put more than $116 billion into people’s pockets in 2009 and 2010. Yet, a Times poll found that fewer than 10 percent of those surveyed had any clue. Remarkably, fully one-third thought their taxes went up—even though the actual number was about zero. 1

The Obama administration’s theory that spreading tax cuts through withholding would be more effective stimulus than lump-sum checks was probably right on the economics, but politically, it was a disaster. It may have helped Republicans win the House in 2010, even as voters were receiving tax cuts they didn’t know about.

Housing reform faces the same problem. If rents grow 2% instead of 6%, will anyone connect it to a zoning change from eight years earlier? I doubt it.

Housing Reform’s Invisible Results

Minneapolis is the poster child for YIMBY success.

Minneapolis’ success in building new apartments has enabled the city to substantially add to its housing supply and keep rent growth low. From 2017 to 2022, Minneapolis increased its housing stock by 12% while rents grew by just 1%. Over the same period, the rest of Minnesota added only 4% to its housing stock while rents went up by 14%.2

But notice the timeline. The Minneapolis 2040 plan was adopted in late 2018, with major zoning changes taking effect in 2020. The dramatic results took years to materialize, and this is the optimistic case.

Paradoxically, abundance reforms can actually make things worse before they get better. For example, when Chicago implemented targeted up-zoning from 2013-2015, there was no discernible increase in housing construction in the impacted areas. Even more concerning, property values actually increased potentially due to developer anticipation of projects that have yet to materialize.3

Over time, housing reform will make people better off and it’s the right thing to do, but history suggests the median voter will take the results for granted.

Unlocking the Housing Shadow Supply

I heard about a family who recently moved from NYC to Buffalo because they had a remote job situation that enabled it. The median house price there is around $200K4 and houses sell for around $100K in certain neighborhoods. Ample underutilized housing exist across the country, but under today’s setup, the commutes are unworkable. Remote work is a structural reform of the housing market. The crisis is a location mismatch, not a lack of housing. Workers are being artificially forced to compete for a limited pool of housing within a commutable radius.

By unbundling work from location, we effectively bring a massive “shadow supply” of housing onto the market. When a worker moves from a high-demand city center to an under-utilized home in a smaller town or a distant suburb, they aren’t just improving their own life; they are de-pressurizing the urban core.

Counter-intuitively, this is the best thing that could happen to “on-site” workers. When the “laptop class” stops outbidding the nurse, the firefighter, and the service worker for the apartment next to the hospital, rents for essential workers actually stand a chance of falling. Remote work relieves rent competition.

The Salience of Remote Work

If the goal of the Abundance movement is to deliver visible improvements to people’s lives on a timeline that matters, we need to look beyond the slow churn of construction. When a worker gains ten hours a week back by skipping a soul-crushing commute, they notice. When a parent no longer has to juggle childcare drop-off, rush to an office miles away, and then pay for extra after-care just to cover time spent sitting in traffic, they feel it. This kind of flexibility is the most immediate housing policy we have. It delivers direct, daily relief to working families in a way new construction cannot, because it changes how people live right now, not years later.

The Real Force Behind Return-to-Office

Return-to-office mandates are often framed as business decisions: executives wanting collaboration, culture, or accountability. But there’s another force at play that rarely gets named: political pressure from city governments.

Cities like New York have built their fiscal models around commuter presence. Commercial property taxes, transit revenue, and downtown retail sales tax all depend on office buildings being full. When remote work emptied those buildings, it didn’t just hurt landlords, it threatened municipal budgets.

City officials have responded by pressuring employers to bring workers back. Former Mayor Adams was explicit about this, calling remote work a problem to be solved and urging companies to return to offices5. This isn’t subtle. It’s a policy priority.

Employers, in turn, face a choice. Some genuinely believe in-office work is more productive, and if that’s true, they’ll pay the premium in higher salaries or lost talent and the market will sort it out. But others are responding to political pressure, not business logic. They’re making decisions about where their employees spend 40 hours a week based partly on what city officials are asking of them, and the cost falls most heavily on families.

When a parent is forced back into a commute, that’s hours lost to childcare shuffling, missed dinners, and the logistical grind that remote work briefly relieved. It’s the single parent who now needs extended aftercare. It’s the father who used to do school pickup and no longer can. It’s the couple who moved further out for space and now faces an impossible commute.

These families are being used to plug the holes in a municipal revenue model.

An Opportunity for Pro-Family Politics

Some of the same officials who claim to support working families are actively supporting policies that make family life harder. The same governments that worry about declining birth rates and childcare costs are pressuring employers to impose commutes that exacerbate both problems.

This is an opportunity.

Pro-family advocates should recognize remote work as one of the most tangible, immediate quality-of-life improvements available to working parents. Unlike housing construction, which takes years, or tax credits, which disappear into paychecks unnoticed, flexibility is felt daily. It’s the kind of visible, concrete benefit that builds political loyalty.

A political coalition that positioned itself as defending family time against the demands of city budgets built around commuter dependence would be offering something real. Not a promise of future affordability, but an immediate improvement in daily life.

A City of Choice

As a resident of New York City, I love the density, the community, and the culture of urban life. But a city propped up by captive commuters is a fragile one.

The Abundance agenda should strive for a city of choice. If we stop forcing people to live in NYC just for a desk, we make it possible for the people who actually want to be here to afford it. By allowing the “unwilling participants” to exit the market, we create a more resilient, affordable, and vibrant urban core.

Housing reform is a noble long-term goal, but it needs a political shield. Remote work provides that shield. It delivers the immediate, visible “win” that buys a movement the time it needs to build the future.

When I started building a forecasting bot for the Metaculus AI Benchmark, I was spending my own money. That meant using smaller, cheaper models and squeezing out every bit of signal I could through methodology.

I built a structured pipeline that approximated how a human forecaster would approach a question. On my initial tests, this process-driven approach outperformed the raw model by a healthy margin, and it felt like I was really onto something.

Then Metaculus gave me API credits, and suddenly I could afford frontier models.

I ran my pipeline against these frontier models, expecting my clever methodology to stack on top of their superior reasoning. Instead, I crashed into what so many before me have experienced: the Bitter Lesson.

The Bitter Lesson

The Bitter Lesson is Rich Sutton’s observation that in AI, methods that scale with computation tend to outperform approaches that rely on human-designed structure or domain knowledge. My forecasting pipeline was my human-designed structure, and it didn’t add anything to what the frontier models already provided.

What the Data Showed

My process-driven pipeline, which had shown promise with smaller models, didn’t add noticeable value with frontier models. In my testing, a better use of tokens was simply polling larger models more times to build a larger ensemble, rather than spending tokens on classification, routing, or decomposition.

Why This Happens

My best guess is that, with weaker models, the pipeline helped because it imposed structure on reasoning that the model didn’t reliably produce on its own. It was scaffolding for limited capability.

Frontier models don’t seem to get much value from this scaffolding. They can reason through base rates, consider multiple scenarios, and synthesize information without being explicitly told to. The pipeline was just consuming tokens that could have been better spent on additional polling.

What I’m Doing Now

I’ve simplified. The elaborate pipeline has been replaced with more polling, and I have new theories I’m testing against the crowd. My testing suggests there may be a way to elicit latent knowledge from models that meaningfully improves forecasts, but I need more data before I’m confident the effect is real.

Testing and Validation

I used Metaculus community probabilities as a calibration benchmark. A baseline frontier model might correlate at r ≈ 0.65, while aggregating multiple frontier models correlated at r ≈ 0.74. Including the pipeline did not provide additional uplift over ensembling.

While not perfect, this provided a useful sanity check. This kind of empirical validation is what makes forecasting bots useful. We can actually test what works instead of arguing about theory.

The Lesson

I started this project trying to stretch small models with clever methods, but it turns out the winning approach was just using better ones. Sometimes the boring answer is the one that actually scales. For now, with frontier models, keep it simple: ensemble aggressively and validate empirically.

This is the fifth post in a series on building an AI forecasting bot. In previous posts, I covered the Broken Leg check (short-circuiting on breaking news) and classification and method selection (routing questions to the right forecasting approach) and decay functions (updating as time runs out).

Forecasting failures are often obvious in hindsight. A lot of them could have been caught by a brief moment of structured skepticism. Conceptually, there are two parts: a pre-mortem and a sanity check. They are related, but they catch different classes of mistakes. The pre-mortem asks “what could go wrong?” while the sanity check ensures the forecast doesn’t violate basic reasoning.

What Could Go Wrong?

The pre-mortem assumes the forecast is wrong and asks why.

What plausible unconsidered path could change the outcome?

Consider a question like:

Will Company X release Product Y by December 31?

Suppose the pipeline outputs 65%, based on past delivery timelines, current hiring signals, and executive statements. This may be directionally reasonable, but it embeds a quiet assumption: that the product is released at all, cleanly, under the same name, and under the same definition implied by the question.

In practice, questions like this commonly fail in predictable ways.

Maybe the product is canceled. Maybe the company is acquired invalidating the “Will Company X” portion of the resolution criteria. Maybe the release happens, but in a form that doesn’t meet the specification of the question.

A forecaster might try to account for many of these implicitly, but being explicit makes it less likely that something important will be left out. These adjustments are often modest, but they materially improve the forecast.

Wait, What?

The sanity check asks a different question:

Does this number make sense given what we know about the world?

Sometimes, even for a human, following a sound process can still produce obvious failures.

Consider the question:

Will it snow on July 4th, 2026 in New York City?

Suppose the pipeline outputs 3%.

It is easy to see how this could happen if the bot grabs the wrong base rate. NYC gets snow ~12 days a year ≈ 3.3%, but to anyone with lived experience, this is obviously absurd.

If a forecast implies something wildly inconsistent with ordinary experience, one of its assumptions is probably wrong. The sanity check does its job by catching and correcting obvious problems.

Completing the Pipeline

The pre-mortem and the sanity check serve different roles. The pre-mortem looks for missing paths to failure. The sanity check looks for structural mistakes that make the forecast incoherent even if no single step was obviously wrong.

If either of these fire, the forecast needs a second pass, and possibly a different approach altogether.

Once these checks are complete, the forecast is ready to submit.

This is the fourth post in a series on building an AI forecasting bot. In previous posts, I covered the Broken Leg check (short-circuiting on breaking news) and classification and method selection (routing questions to the right forecasting approach).

Now we’re in the final stages of the pipeline: taking a raw method output and adjusting it to account for how probability decays over time.

The Problem

Forecasting methods typically output a single probability: “There’s a 30% chance this happens before the deadline.” But that number hides an important question: how is that probability distributed across the time window?

A 30% chance spread evenly across two years is different from a 30% chance concentrated in the final month. If we’re six months in and nothing has happened, these two forecasts should update very differently.

This is where cumulative probability comes in. The method output represents the total probability that the event occurs somewhere in the window. As time passes without the event occurring, we consume portions of that probability. The decay function tracks how much probability remains based on where we are in the window and where the probability mass was concentrated.

Decay Pattern

The decay classification portion of the pipeline buckets questions into a decay pattern based on how probability is distributed over time:

Constant Hazard

Some events could happen at any time with roughly equal probability. Earthquakes might fall into this classification. The chance of a major earthquake in Seattle next month is about the same whether we’re asking in January or December.

For constant hazard events, probability decays at a steady rate as time passes. Each month that passes without the event consumes a proportional slice of the cumulative probability. If 25% of your time window has elapsed without the event, you’ve consumed roughly 25% of the original probability.

Increasing Hazard

Some events have their probability concentrated near the deadline. The classic example is a debt ceiling negotiation. Neither side wants to blink early since maximum leverage comes from holding out. Maybe 10% of the probability is spread across the first eleven months, 35% is in the final two weeks excluding the last day, and 55% is on the last day before default.

For increasing hazard events, probability decays very slowly at first. Early time passing without the event is expected—you’re not consuming much of the cumulative probability because there wasn’t much probability there to begin with. If eleven months pass without a deal, you’ve only burned through 10% of the total probability mass. As the deadline approaches and you enter the high-probability window, the decay rate accelerates.

Decreasing Hazard

Some events have their probability concentrated early in the window. The classic example is a ceasefire. If it’s going to collapse, it’s most likely to collapse in the first few days or weeks. The probability of collapse is front-loaded.

For decreasing hazard events, probability decays rapidly at first, then slows. Early survival is expensive. It burns through the front-loaded probability mass quickly. If a ceasefire holds for six months, most of the “collapse probability” has already been consumed without the event occurring. You’re now in the long tail where collapse was always unlikely, so further time passing costs you less.

Event-Driven Hazard

For completeness, I’ll mention another decay type that I use in forecasting, but didn’t implement due to complications. Sometimes, key events with a lot of probability mass happen at times that are not particularly close to either extreme.

An example might be a previously scheduled military exercise that could quickly pivot to an attack. If no attack materializes by the time the exercise is over and no further exercises are planned, it might be appropriate to expect the probability to decay rapidly afterward. As I mentioned, I did not implement this since it would be too easy for an AI to over-update on events.

How It Works

The decay analysis takes the method output and the hazard pattern, then calculates how much cumulative probability remains given how much time has elapsed.

Conceptually:

1. The method outputs a total probability P for the entire window

2. The hazard pattern defines how P is distributed across time

3. Given elapsed time t, we calculate how much of P has been “consumed” without the event occurring

4. The remaining probability is our updated estimate

For constant hazard, this is straightforward: linear decay. For increasing and decreasing hazard, the decay follows the shape of the probability distribution. Back-loaded distributions decay slowly then quickly; front-loaded distributions decay quickly then slowly.

Why It Matters

Decay functions matter most for questions with long time horizons where we’re partway through the window. If a question resolves in two weeks, there’s not much room for temporal dynamics to play out. If a question resolves in three years and we’re eighteen months in, the decay adjustment can meaningfully shift the estimate.

Consider a debt ceiling question at the start of the year with a December deadline. Your method might output 70% chance of a last-minute deal. In March, that 70% should barely have decayed as almost all the probability mass is still ahead of you. In November, it should still be close to 70%, because you’re just now entering the window where deals actually happen. Only if December arrives and passes without a deal does the probability finally collapse.

Contrast this with a ceasefire question. If your method outputs a 40% chance of collapse in the first year, and six months pass with the ceasefire holding, you’ve burned through most of that probability. Your updated estimate might be 10% or lower, not because new events have unfolded, but because the high-risk period has passed without incident.

Classification

How does the bot know which pattern applies? It’s part of the classification step. The LLM looks at the question structure and identifies cues:

• Negotiation dynamics → increasing hazard

• Random or memoryless processes → constant hazard

• Stability as evidence of equilibrium → decreasing hazard

Most questions default to constant hazard unless there’s clear structure suggesting otherwise. It’s the most conservative assumption. You’re not claiming to know something about temporal dynamics that you might be wrong about.

The Limits

Decay functions are a refinement, not a revolution. They adjust estimates at the margins based on temporal structure.

And the classification can be wrong. Some questions look like constant hazard but have hidden deadline pressure. Some look like decreasing hazard but have sleeper risks that compound over time. The decay function is only as good as the pattern recognition that assigns it.

As with everything in this pipeline, I’m tracking whether decay adjustments improve calibration.

This is the third post in a series on building an AI forecasting bot. In the first post, I argued that forecasting bots represent an opportunity to test methods empirically at scale. In the second post, I got the bot running and cut costs by ~30x to enable experimentation and rapid prototyping. In the third post, I covered the Broken Leg check, the first step in the pipeline, and a short-circuit for questions already resolved by breaking news or overwhelmingly likely to fall on a particular side.

Now we get to the core of the pipeline: figuring out what kind of question you’re dealing with and applying the right method.

Why Classification Matters

Not all forecasting questions are the same. “Will Bitcoin drop 25% from its peak before 2030?” is fundamentally different from “Will the US Constitution be amended to allow a third presidential term by 2028?” The first has historical data you can extrapolate from. The second is asking about a rare, discrete event with no direct precedent.

A good human forecaster recognizes this immediately and adjusts their approach. They don’t apply the same mental model to every question. Neither should a bot.

In real life, I might have 50 ways of classifying and forecasting a question, but I don’t have time to encode them all into a bot. Fortunately, I think 4 approaches capture ~80% of the value.

My pipeline classifies each question into one of four types, then routes it to a specialized method:

• BASE_RATE: Questions about events with identifiable reference classes

• TIME_SERIES: Questions about quantities with historical trends

• CONDITIONAL_CHAIN: Questions requiring multiple independent conditions

• NOVEL_EVENT: Questions about unprecedented situations

I’m not the only one taking this approach. On the Metaculus AIB Resources page, mmBot describes a similar strategy:

My bot first does research planning by decomposing the question and generating targeted search queries, scraping and summarizing the most relevant articles (llm-driven). It then classifies the question and routes it (also llm-driven) to one of several specialized context pipelines (eg. stock, sport, election, etc). Finally the optimized prompt and context is used to prompt an llm for a prediction several times which are then ensembled into the final forecast.

The intuition is the same: different question types benefit from different approaches. The classification step costs a few tokens but determines the appropriate method.

The Four Methods

Base Rate

Most forecasting questions are base rate questions in disguise. “Will X happen?” is really asking “How often do things like X happen, and is there any reason to think this case is different?”

The method:

1. Identify the reference class: what category does this event belong to?

2. Find the historical base rate for that class

3. Adjust for specific conditions that make this case more or less likely

4. Update based on current evidence

For example: “Will it snow in New York City on December 25th, 2026?”

Reference class: December 25ths in New York City. Historical rate: about 20%1 of Christmas Days in NYC have measurable snowfall since records began. Adjustment: current climate trends (slightly warming), La Niña/El Niño conditions for that year, whether there’s existing snow cover. The base rate anchors the estimate; the adjustments move it.

The danger is picking the wrong reference class. “Christmas Days in NYC” gives you one rate. “December days in NYC” gives you another. The skill is in choosing the class that’s most predictive for this specific question.

Time Series

Some questions have quantitative historical data that can be projected forward. “Will Bitcoin drop 25% from its peak before 2030?” is asking about a price trajectory with years of history.

The method:

1. Extract relevant historical parameters (volatility, trend, mean reversion)

2. Build a simple model of the process

3. Run Monte Carlo simulations forward to the resolution date

4. Calculate the probability of crossing the threshold

This works well when the underlying process is relatively stable and you have enough history to estimate its parameters. It works poorly when the process itself might change or when you’re extrapolating far beyond your data.

For crypto prices, the historical volatility is high enough that a 25% drawdown over a is very plausible and the time series method captures this by simulating many possible paths and counting how often they cross the threshold.

Conditional Chain

Some questions are really asking about a conjunction of independent events. Consider: “Will JD Vance be president on January 20, 2029?”

This isn’t a single event, it’s a chain of conditions that all need to happen:

1. Will he run for president?

2. Will he win the Republican primary?

3. Will the general election happen as scheduled?

4. Will he win the general election?

5. Will nothing prevent him from being sworn in?

Each step has its own probability conditional on the previous step which can be multiplied together to produce the forecast. Even if each step is reasonably likely—say 92%, 66%, 98%, 45%, 99%—the product is about 27%. Much lower than any individual step.

The method:

1. Decompose the question into necessary conditions

2. Estimate each condition’s probability (often using base rates)

3. Assess independence and adjust for correlation

4. Multiply through for the final estimate

This method can be brutally deflationary. Humans chronically underestimate how unlikely conjunctions are. A chain of “probably” events quickly becomes “probably not.”

The skill is in the decomposition. Too few steps and you miss crucial dependencies. Too many and you’re multiplying noise. You want the load-bearing conditions that actually determine the outcome.

Novel Event

Some questions have no reference class and no historical data. “Will a U.S. or U.S.-ally satellite be permanently disabled by another country or organization before January 1, 2027?” is asking about something that hasn’t happened yet (at least not in a confirmed, public way).

For genuinely novel events, I use a Laplace prior. The intuition: if something hasn’t happened in N attempts, your best estimate for it happening next time is 1/(N+2). This comes from Laplace’s rule of succession, a principled way to reason about events with zero observations.

The method:

1. Pick a reasonable starting point for when the event could have begun happening. For satellite attacks, we might select 1995 since that’s when GPS became fully operational and space infrastructure became a meaningful military target. That gives N=30 years of opportunities with zero confirmed events.

2. Calculate the per-period probability: 1/(N+2) = 1/32 ≈ 3.1% per year.

3. Apply it over the remaining window. With about 1 year until the deadline, the cumulative probability is 1−(1−1/32)^1 ≈ 3.1%.

4. Adjust for evidence not captured in the base calculation. There are more satellites now than in 1995. There have been suspected cyber attacks that might have temporarily disabled satellites. Tail risks like Russian escalation or Taiwan conflicts could change the calculus. Nudge the estimate as appropriate.

The key insight is that “unprecedented” doesn’t mean “impossible.” It means you should be uncertain and should flag that uncertainty clearly.

Classification in Practice

The classifier is a prompted LLM call that takes the question text and outputs one of the four categories plus a brief justification. It’s fast, reasonably accurate, and provides real uplift for forecasts in empirical results so far. Better instructions let us get away with using smaller, cheaper models without loosing too much in performance.

The tricky cases are questions that could plausibly fit multiple categories.

“Will the US recognize Taiwan before 2029?”

This could be:

• BASE_RATE: How often do countries make major diplomatic recognition changes?

• CONDITIONAL_CHAIN: Requires US decision + willingness to accept China’s response + no reversal

• NOVEL_EVENT: The US-Taiwan-China situation is historically unique

In ambiguous cases, the classifier picks the method most likely to produce a calibrated estimate. Usually that means defaulting to base rate with adjustments, since that’s the most robust approach when you’re uncertain about structure.

Why Split It Out

Having the methods separated lets me track what works and what doesn’t. Each question in my dataset records which classification it received, how the method-specific logic processed it, and what probability came out. As questions resolve, I’ll be able to answer:

• Which classifications are most accurate?

• Are certain methods systematically overconfident or underconfident?

• Do some question types benefit more from structured reasoning than others?

• Where should I invest effort in improving the pipeline?

This is the whole point of running at scale. Instead of debating whether conditional decomposition “should” work in theory, I can measure whether it actually works in practice. The tournament generates the feedback loop.

This post is part of a series on building an AI forecasting bot. In my last post, I described getting the bot running and cutting costs by almost 30x. Now I want to start walking through the pipeline itself.

The first step mirrors what a skilled human forecaster would do: check if you can skip the work entirely. Not only does this save time for a human forecaster, it saves computation and therefore money if a bot does it correctly.

The Broken Leg Rule

The concept comes from psychologist Paul Meehl, who spent his career studying when statistical models outperform human judgment. His finding was consistent and humbling: simple actuarial formulas beat expert intuition in domain after domain, but Meehl identified an exception.

Suppose you’re predicting whether someone will go dancing on Friday night. Your statistical model, trained on their past behavior, says 99%. But you happen to know they broke their leg this morning. You should override the model.

This “broken leg” represents information so decisive that it renders the baseline analysis irrelevant. The model can’t account for it because it’s rare, specific, and outside the training distribution. But when you have it, you should use it.

Meehl’s point wasn’t that intuition beats statistics. It was the opposite: trust the model almost always, but recognize the rare cases where you have information the model doesn’t.

Why Check First

For a forecasting bot with a complicated pipeline, there’s a practical reason to check for broken legs before doing anything else.

If the question is “Will Nicolás Maduro cease to be president of Venezuela during 2026?” and the news says U.S. forces just captured him and flew him to New York, you don’t need base rates. You don’t need time series analysis. You don’t need to decompose the question into conditional chains. The answer is already overwhelmingly clear.

A human forecaster would recognize this immediately and move on. The bot should do the same. Spending tokens on a full analysis when the outcome is already determined is wasted computation.

The Broken Leg check is the first step in my pipeline because it’s the cheapest way to potentially avoid all subsequent steps. A quick news lookup costs a fraction of what the full pipeline costs. If it fires correctly, you save everything downstream.

The Danger

Here’s the problem: being confidently wrong is catastrophic in a forecasting tournament.

Proper scoring punishes confidence asymmetrically. A broken leg check that fires incorrectly can sink your entire tournament. One blowout erases the gains from dozens of correct shortcuts.

I’ve seen human forecasters make this mistake. A headline feels definitive. Congress passes a law. A major announcement drops. The forecaster updates hard, confident they’ve found a broken leg. But the resolution criteria are more nuanced than the headline suggests, and the “decisive” news turns out to be merely suggestive.

The instinct to update on breaking news is correct. The instinct to update all the way is dangerous.

The Trade-off

So why include the check at all?

Because when it works, it’s extremely valuable. Not just for accuracy but for efficiency. You’ve spent minimal resources to correctly skip an expensive pipeline. Those saved resources compound across questions.

The trade-off is straightforward: spend a little up front to potentially save a lot. But only if your broken leg detector is reliable enough that the expected savings exceed the expected cost of the occasional blowout.

This is an empirical question. You can’t know the answer in advance. You have to run the bot, track when the check fires, see how often it’s right, and calculate whether you’re coming out ahead.

Right now, I’m still gathering data. The check is implemented and I’m monitoring its performance. If the blowouts start outweighing the savings, I’ll tune it, add confidence dampening, improve the news filtering, or disable it entirely until I can make it more robust.

Implementation

The actual implementation is simple. Before running the full pipeline, the bot queries a news API with the question text. If the returned summary contains information that overwhelmingly resolves the question in one direction, output a high-confidence probability and stop.

The hard part isn’t the code. It’s defining “overwhelmingly resolves.” That’s where the judgment lives, and where the misfires come from.

A president being captured and jailed? Overwhelmingly resolves “will they cease to be president.” A law being passed requiring document release? Maybe. Depends on enforcement, interpretation, timing, and exactly what the resolution criteria specify.

The more I can teach the bot to distinguish between these cases, the more value the Broken Leg check provides. That’s an ongoing refinement.

What’s Next

The Broken Leg check is step zero—the escape hatch. Most questions won’t trigger it, and the bot will proceed to the main pipeline.

In the next post, I’ll cover what happens when you don’t have a broken leg: how the bot classifies questions and selects the appropriate forecasting method.

My forecasting bot is now live in the Metaculus AI Forecasting Benchmark.

In my last post, I argued that this tournament represents forecasting’s “ImageNet moment” - a chance to test methods empirically at scale. Now the work begins.

The Forecast Pipeline

Most AI forecasting bots take a similar approach: feed the question to an LLM, maybe do some web search, ask for a probability. Some get fancy with multi-agent debate or ensemble averaging across models.

My bot runs each question through a structured pipeline: a sequence of steps designed to mimic how good human forecasters actually think. The pipeline details will come in future posts as I reveal each stage, but the flow mirrors how I operate as a superforecaster.

Codifying it into a reproducible system that can be tested and refined? That’s the experiment.

Top Priority: Reduce Costs

I applied for LLM credits from Metaculus, but I haven’t heard back yet and I’m currently paying out of pocket for LLM usage (AskNews generously provided free API access for news retrieval over the weekend - thank you!).

To make this work for my purposes, I need lots of iterations. The tournament has hundreds of questions. That requires running the bot many times. And running LLMs many times gets expensive fast. I want to test different approaches, compare results, and figure out what actually works. In practice, that means before I could iterate on methodology, I needed to make iteration affordable.

The unmodified template bot cost:

$0.109 per question

After my cost-reduction updates:

$0.004 per question

That’s a 27x reduction.

At the old rate, running the bot across 500 questions would cost ~$55. Now it’s ~$2. I can test dozens of different approaches for what one used to cost.

The Trade-off

I expect a performance hit from these changes. Cheaper models, fewer tokens, less elaborate reasoning. That’s fine for now.

The goal at this stage isn’t to win, it’s to learn. Which parts of the pipeline matter most? Where does additional compute actually help? What’s signal versus noise?

Once I understand what works, I can selectively add compute back where it counts.

What’s Next

The bot is running. Data is accumulating. Over the coming weeks, I'll share results, break down each pipeline step, and adjust based on evidence.

This is forecasting about forecasting using the tournament as a feedback loop to refine the methodology itself. The first real test: does a structured pipeline, even a cheap one, beat naive approaches?

We’ll find out.

Since its beginning, forecasting has been more craft than science. There were theories about what works (e.g. base rates) but testing them was slow, expensive, and small-scale. Forecasters could disagree about method and never get to a resolution because the feedback loops were long and sample sizes were too small.

That’s changing.

Forecasting bots are automated systems that take questions - “Will X happen by Y date?” - and output probability estimates. They can run 24/7, answer hundreds of questions, and get scored against outcomes. The Metaculus AI Benchmark is now running tournaments with hundreds of questions and real-time feedback. We can finally test forecasting methodologies empirically at scale.

This is forecasting’s ImageNet moment. ImageNet was the dataset that transformed computer vision from an academic backwater into the foundation of modern AI. Before ImageNet, researchers argued about which approaches to image recognition were best. After ImageNet, they could just run experiments and see.

The same thing is about to happen to forecasting. We’re going to learn what actually works.

I’ve written before about simple forecasting instructions that get you 80% of the way there, but building a bot that performs at the frontier requires going deeper. The simple heuristics are table stakes.

I’m building a bot. And I’m going to reveal my methodology in a series of posts. After the tournament ends, I’ll release everything.

The field is about to take a leap forward. Might as well document it in real time.

A driver with 5,000 rides and a 4.9 rating on Uber? Counts for nothing on Lyft.
A host with a 5 star rating on Airbnb? Try convincing guests on VRBO to book without offering a steep discount. A seller with 800 successful transactions on Ebay? Good luck when you try to break in to the Amazon marketplace.

It’s not a technical limitation, It’s a business model.

Platforms trap you because rebuilding reputation is costly. Each move to a new platform applies a reputation tax. You start at zero, and the platform keeps your identity in a walled garden.

In theory, cryptographic identity verification could address this, but, until now, the infrastructure has remained largely inaccessible.

Cipherblock.io changes this by making reputation a portable, verifiable identity that you control.

Portable Trust

In the abstract

• You carry a single keypair that represents your public identity.

• Anyone can publish an attestation about you that is digitally signed, timestamped, and published publicly.

• Any platform can verify an attestation in constant time without asking us or trusting a third party.

In practical terms

• A rideshare driver with 5,000 rides asks riders to sign attestations

• Each attestation is cryptographically signed and timestamped

• When joining another, the driver presents these attestations and users can see that these attestations are very likely coming from actual people and not fake accounts because the software can typically tie them back to someone they know or the signals are too costly to fake

• The platform verifies the signatures instantly no phone calls, no manual review

• The driver starts with established credibility, not from zero

Reputation stops being a private resource and becomes a set of verifiable claims you can take anywhere.

Technical Foundation

Everything runs in the browser meaning no downloads and no setup required:

• Generate your keypair client-side in seconds (private key never transmitted to any server)

• Anyone can run an operator that records attestations

• Operators maintain interoperable chains and compete for users without lock-in

• Users store signed receipts as cryptographic proof; even if an operator disappears, you keep the attestations

• Sites can integrate cipherblock for identity management with a few lines of code (an npm package)

• Open source and verifiable

The infrastructure is live at cipherblock.io. Read the technical specification to understand how it works.

Why It Just Might Work

• New entrants can bootstrap trust networks without years of building reputation systems

• Smaller platforms can compete with incumbents on service quality rather than network effects

• Regulatory pressure for data portability is increasing and users increasingly value platforms that respect their autonomy

Why It Might Not Work

• It introduces trade-offs that many people are unaccustomed to and might therefore hate

• Depends on network effects as well, if there aren’t enough adopters no one will want to support it perpetuating the problem

• It depends on someone caring about this enough to do something

Get Started

**Important: This is a functional prototype and active research project.**

The infrastructure works and is live at cipherblock.io, but expect rough edges, breaking changes, and ongoing experimentation. Think of this as an invitation to explore the concept, not production-ready software.

For Users: Build Your Portable Identity

1. Visit cipherblock.io - Generate your keypair in seconds, entirely in your browser

2. Grab tokens while you’re there or if the faucet is out contact me directly and I’ll give you some.

3. Request attestations - Ask people you’ve worked with to provide signed ratings

4. Store your receipts - Save the signed attestations locally as cryptographic proof

5. Take them anywhere - Present your portable reputation when joining new platforms or applying for opportunities

Your identity. Your reputation. Your control.

Even if every housing reform went into effect tomorrow, construction timelines mean it could be years before we see meaningful improvements in affordability or availability. To improve quality of life and opportunity in the near term, we need high-leverage, low-friction interventions that reshape how people live and move right now.

Mobility is the highest leverage near-term option we have.

Mobility improvements offer immediate and high-leverage benefits, but are just one component of addressing our housing affordability crisis. Mobility enhancements can deliver tangible benefits while housing developments are still navigating approval processes.

What Abundance Means

By "abundance," I’m referring to a material prosperity that enables greater life choices and opportunities. In its simplest form, abundance means having enough affordable housing, efficient transportation options, and economic opportunities so that people can thrive rather than merely survive. This concept explains why many are leaving traditionally progressive states like New York and California for places like Texas and Florida, where housing costs are lower and day-to-day living is more affordable. This exodus has political consequences: as population shifts, so does electoral representation. If New York wants to preserve its importance, we need tangible policies that improve material well-being, starting with mobility solutions that can be implemented quickly while we work on longer-term housing reforms. When people can access jobs, education, and amenities affordably and efficiently, we create the conditions for genuine abundance.

Mobility Improves Affordability

Housing affordability isn't just a function of how many units exist, it's also about where those units are, and how easily people can access jobs, schools, services, and social networks from them. Mobility is a force multiplier: when people can get where they need to go quickly, reliably, and safely, it effectively increases the supply of livable housing, even if the number of units stays the same.

There are three core ways transportation improves affordability:

Expanding Access to Undervalued Neighborhoods

Across New York, thousands of housing units remain underutilized simply because they're isolated by inadequate transit options. A faster, more flexible transportation system opens up these areas to new demand and investment, raising utility without having to wait for new supply to come online. It also has the added bonus of incentivizing further investment improving the total housing supply.

Reducing the Combined Housing-Transportation Cost Burden

When reliable transportation options exist, families can make strategic trade-offs between housing and transportation costs. Improved mobility allows households to access more affordable housing farther from city centers without sacrificing job opportunities or dramatically increasing commute times. This shifts the traditional calculus of "drive until you qualify" to a more sustainable model where transportation doesn't consume the savings gained from lower housing costs.

Creating Economic Mobility Without Physical Relocation

Robust transportation networks connect people to a wider range of employment, education, and service opportunities without requiring them to move. This is particularly powerful for lower-income communities where residents can access better jobs and amenities while maintaining crucial social networks, community ties, and cultural connections. Transportation becomes a ladder to opportunity that doesn't require abandoning affordable housing or supportive communities. If better transportation means families have access to better job opportunities, it improves affordability because they have higher incomes.

Solutions for Near-Term Impact

New York already has the tools needed to dramatically improve mobility in the near future and some of these solutions are already underway, but they need our continued support. I’ve tried to highlight solutions that can be implemented within months to years, as opposed to years to decades, providing tangible benefits while longer-term housing reforms take shape.

The focus here is primarily on low-friction improvements that don't require massive capital outlays like building new rail lines or other major infrastructure projects. Instead, these solutions maximize existing resources, repurpose underutilized assets, and implement policy changes that can deliver near-term benefits at relatively modest costs. By prioritizing these high-leverage, low-friction interventions, we can achieve significant mobility improvements without waiting for complex, expensive infrastructure projects to complete their lengthy planning and construction cycles.

Expanding Congestion Pricing

Expanding Manhattan's congestion pricing to high-traffic areas in Brooklyn and Queens could reduce congestion while generating revenue for transit improvements. By thoughtfully designing fee structures, we can ensure access while discouraging unnecessary traffic.

Repurposing Street Space for Dedicated Bus Lanes

New York can learn from London's successful transformation of its bus network through strategic repurposing of road space. In 2021, Transport for London (TfL) made 85km of bus lanes on its busiest roads operate 24 hours a day, seven days a week. This bold move significantly improved journey times and service reliability, particularly in central and inner London areas.

The results speak for themselves: bus service became more predictable and faster, making it an attractive alternative to driving. As Louise Cheeseman, TfL's Director of Buses, noted, "Shorter journey times will help deliver a more reliable bus network," which in turn helps "make buses an attractive travel option." Most importantly, these changes created a positive feedback loop: more reliable service attracted more riders, which justified further service improvements, creating a virtuous cycle of transit growth.

This stands in stark contrast to the current situation in New York, where bus unreliability has decimated ridership. Most New Yorkers don't consider buses a viable transportation option precisely because of unpredictable arrival times and crawling speeds when stuck in traffic. Nobody wants to wait 20 minutes for a bus that then inches along in congestion, turning what should be a quick trip into a frustrating ordeal.

By converting underutilized street parking into dedicated bus lanes, we can break this cycle of decline. Continuous, camera-enforced bus lanes that operate 24/7 could be implemented relatively quickly by repurposing existing infrastructure, delivering immediate benefits to hundreds of thousands of New Yorkers daily. The evidence from London1 shows that these measures can dramatically improve service without significantly impacting other traffic, providing a rare win-win in urban transportation policy.

Demand-Based Parking Reform

Strategic adjustment of parking pricing based on demand has proven remarkably effective at reducing traffic congestion in urban areas. San Francisco's pioneering SFpark program demonstrates the power of this approach. By implementing demand-responsive pricing at parking meters, San Francisco reduced "cruising for parking" — drivers circling blocks looking for spaces — by 43%, which translated to a 30% decrease in vehicle miles traveled in SFpark areas.

The program works through a simple application of supply and demand economics: when parking demand is high on a particular block, prices increase slightly ( e.g. $0.25 per hour); when demand is low, prices decrease by the same amount. This creates a natural distribution of parking across neighborhoods while ensuring that spaces remain available on every block.

Contrary to initial concerns about increasing costs, the program actually resulted in lower average parking rates (down 4% for on-street parking and 12% for city-owned garages) while simultaneously increasing business activity. Sales tax revenue rose 35% in SFpark areas compared to less than 20% in other parts of the city.

By implementing similar demand-responsive parking policies in New York, we could immediately reduce the estimated 30% of traffic caused by drivers searching for parking, improving mobility for everyone while supporting local businesses and reducing emissions.2

Repurposing Street Space for Bike Lanes

New York has made significant progress in expanding its bike lane network, but too many routes remain disconnected, unprotected, or both. By strategically converting misallocated street space into protected bike lanes, we can create a comprehensive, safe cycling network that serves both commuters and recreational riders.

Protected bike lanes have been shown to substantially increase cycling volumes within just months of installation. This dramatic growth occurs because many potential cyclists cite safety concerns as their primary reason for not biking. When people feel safe, they ride.

My own experience confirms this. Since protected bike lanes were installed connecting Downtown Brooklyn to Williamsburg, I've made them my preferred mode of transportation during much of the year. While driving this route might take 17 minutes on a good day, it's wildly unpredictable – today it took 45 minutes (it was rainy so I foolishly opted to take my chances driving), and I've experienced trips longer than an hour before. By contrast, my bike commute is a consistent ~17 minutes every time, plus I avoid the hassle of finding parking. This reliability is why I often choose cycling over other modes even in winter when protected lanes are available.

Beyond just increasing cycling, protected bike lanes benefit the entire transportation ecosystem. They reduce injuries for all road users (not just cyclists), decrease traffic congestion by encouraging mode shift, and improve business for local retailers along bike routes. Many neighborhoods have seen retail vitality increase after protected bike infrastructure was installed.3

Implementation can be remarkably swift - a basic protected bike lane can be created using paint and flexible bollards in days or weeks rather than months. More permanent versions with concrete barriers can follow as funding permits, but the immediate safety and mobility benefits begin as soon as the lanes are marked.

With thousands of miles of streets in New York City, dedicating just a fraction of this space to safe, protected bike infrastructure could create an accessible network.

Embracing Technological Innovation

Creating a clear regulatory framework for robotaxis and other emerging mobility technologies can accelerate their safe deployment. These services can fill gaps in the transit network, reduce car ownership needs, and free up valuable urban space currently dedicated to parking. While full-scale deployment will take time, establishing the groundwork now ensures these benefits arrive sooner.

Improving Transit Information Accessibility

New York's public transit system, while extensive, often fails at the most basic function of helping riders navigate with confidence. These informational barriers create unnecessary anxiety and discourage transit use, even among regular riders.

Anyone who has tried to navigate New York's subways has experienced the frustration of missing or inadequate wayfinding tools. Maps are scarce throughout stations, making quick orientation difficult. While other global metro systems prominently display route maps in multiple locations within each car and throughout stations, New York riders often find themselves straining to glimpse a distant map or relying on smartphone apps – assuming they have service underground.

Compare this to the Bay Area's BART system, where airport-bound trains are clearly designated with distinct signage. In New York, the train to JFK is indistinguishable from the other on the same line, with no special markings. For visitors and residents alike, this creates needless uncertainty about whether they're heading to their flight or somewhere entirely different.

The critical function of announcing upcoming stops is rendered nearly useless by poor audio quality throughout the system. Garbled, unintelligible announcements compete with the train's noise, leading riders to miss their stops entirely – especially when crowding prevents visual confirmation of location. This basic communication failure creates anxiety for riders who must remain hyper-vigilant rather than using travel time productively.

Perhaps most frustrating is the unpredictable nature of service changes and skip-stop patterns. Riders can never fully trust that their train will make all expected stops, forcing them to constantly monitor announcements (which, as noted, are often unintelligible) rather than using transit time productively for reading, working, or resting. This unpredictability erodes confidence in the system and discourages ridership among those who have alternatives.

Implementing clear, consistent, and accessible information systems would dramatically improve the rider experience at relatively low cost compared to infrastructure changes. Digital displays in every car showing upcoming stops, clear airport route differentiation, improved audio systems, and real-time service alerts visible throughout stations would transform how New Yorkers interact with their transit system. These improvements would make transit more welcoming to occasional users while reducing stress for regular commuters.

By addressing these basic informational shortcomings, we can make transit not just physically accessible, but cognitively accessible as well – a critical step toward encouraging greater ridership and reducing dependency on private vehicles.

Prioritizing Mobility for Immediate Impact

These solutions share three critical advantages over housing reforms alone:

Implementation Speed

These mobility interventions can be deployed within months to years, not decades. London's bus lane transformation and San Francisco's parking reforms demonstrate how quickly well-designed policies can reshape urban mobility patterns.

Leverage Existing Infrastructure

Rather than waiting for new construction, these approaches maximize what we already have. By repurposing street space, optimizing pricing mechanisms, and improving information delivery, we can dramatically increase the utility of our current transportation network.

Proven Success

Each proposed solution builds on documented successes in comparable urban environments. The data from London and San Francisco isn't theoretical - it shows concrete improvements in reliability, accessibility, and even economic vitality.

New York stands at a crossroads. While we must continue pursuing long-term housing solutions, we cannot afford to neglect the immediate mobility crisis affecting millions of residents daily. By implementing these high-leverage mobility interventions today, we create breathing room for housing reforms to take effect, improve quality of life in the near-term, and build momentum toward a more accessible, affordable city.

The path to abundance starts with mobility. Let's take the first steps.

Imagine betting your life savings on a weather forecast, only to discover the meteorologist was relying on a groundhog's shadow instead of analyzing atmospheric data. This is essentially what we do when addressing global existential threats without scrutinizing our underlying world models.

From nuclear war to bioterrorism, from authoritarian regimes to pandemic risks, humanity faces numerous existential threats. Yet we struggle to agree not only on how to address these challenges, but even which ones deserve priority. With our future at stake, we need better methods to distinguish genuine risks from distractions.

The solution may be simpler than we think: systematic forecasting that puts models to the test.

World Models Described

Everyone has a world model, whether they realize it or not. It's an internal representation of reality – the beliefs, assumptions, and mental shortcuts used to make sense of a complex world. Essentially it’s a framework for understanding how the world functions.

When you wake up in the morning and plan your day, you're using your world model. You assume the sun will rise, your lights will come on when you flip the switch, and the world outside your house will exist when you go out the door. These aren't just random guesses – they're based on your understanding of how the world operates.

The problem is, our world models can be flawed, particularly when we don’t get direct feedback on them. They can be based on incomplete information, biased by our personal experiences, or simply outdated. It’s not a big deal when the stakes are low and we find out that while we had no trouble explaining how an airplane can fly normally, we can’t explain how it can fly upside-down. Unfortunately when we're dealing with global challenges, the limitations of our world models become critically important.

Evaluating Competing World Models

The first challenge is that there are a lot of competing world models to consider, so many in fact that evaluating them all would be an intractable problem. In virtually any given field, even if there are many points of consensus, you'll still find areas where there is wild disagreement. This diversity of models reflects the complexity of the world we're trying to understand, but it also makes it difficult to determine which models are most accurate or useful.

On top of that, we face the problem that every now and again the prevailing paradigm is flawed, but gathering the data to overturn it could take a long time to collect. This is challenging because data collection can be inherently difficult, especially for complex global issues, and the feedback loops can be very long, making it hard to quickly validate or invalidate a model. Moreover, existing paradigms often become entrenched not just due to scientific inertia, but also because of ego investments, tribal affiliations within academic or professional communities, and power structures that benefit from the status quo. These social and institutional factors can create significant resistance to new ideas or evidence that challenges established models, further complicating the process of correcting flawed paradigms.

Additionally, even if a model has very good explanatory power, it might still be spurious. Consider the classic example of the correlation between ice cream sales and drowning incidents. A model based on this correlation might appear to have strong predictive power, but it misses the true causal factor (warm weather) that influences both variables.

Evaluating Models with Instrumentalism

It probably goes without saying that some models produce better results than others. Consider this passage from Sapiens.

In 1744, two Presbyterian clergymen in Scotland, Alexander Webster and Robert Wallace, decided to set up a life-insurance fund that would provide pensions for the widows and orphans of dead clergymen. They proposed that each of their church’s ministers would pay a small portion of his income into the fund, which would invest the money. If a minister died, his widow would receive dividends on the fund’s profits. This would allow her to live comfortably for the rest of her life. But to determine how much the ministers had to pay in so that the fund would have enough money to live up to its obligations, Webster and Wallace had to be able to predict how many ministers would die each year, how many widows and orphans they would leave behind, and by how many years the widows would outlive their husbands.

Take note of what the two churchmen did not do. They did not pray to God to reveal the answer. Nor did they search for an answer in the Holy Scriptures or among the works of ancient theologians. Nor did they enter into an abstract philosophical disputation. Being Scots, they were practical types. So they contacted a professor of mathematics from the University of Edinburgh, Colin Maclaurin. The three of them collected data on the ages at which people died and used these to calculate how many ministers were likely to pass away in any given year.

How do we differentiate between competing world models? It depends on the goal, of course. For some, it's about forming a consistent paradigm, for others it's a grab bag of ad hoc rules that work well enough to get by without thinking too hard, and for others it may be more about finding a set of beliefs that allows for minimum conflict within their tribe. For most readers here, however, I suspect it is more about practical utility: using models that help us accurately predict and navigate reality.

This pragmatic approach, known as instrumentalism, judges models not by whether they're "true" in some abstract sense, but by whether they're useful tools for understanding and interacting with the world. Under instrumentalism, the best model is the one that most reliably generates accurate predictions and enables effective intervention.

Consider how we evaluate scientific theories: Newton's laws of motion aren't considered valuable because they represent some perfect, ultimate truth, in fact, we know they break down at quantum scales. They're valuable because they allow us to predict with remarkable accuracy how objects will behave under everyday conditions. Similarly, when evaluating world models for global challenges, we should prioritize predictive power over ideological purity or intuitive appeal.

This is where good forecasters enter the picture. These individuals and teams have empirical track records of successfully predicting future events across diverse domains. Their consistent success isn't theoretical, it's demonstrated repeatedly through verifiable results. While we'll explore the cognitive mechanisms behind their success in a future article, the important point is that their methods work. By identifying who consistently makes accurate predictions, we gain a shortcut to identifying which world models are most effective for understanding reality.

The instrumentalist view liberates us from endless philosophical debates about the "true nature" of reality and focuses our attention on what matters: which models help us anticipate and address real-world problems most effectively. When facing existential risks, this practical focus isn't just intellectually honest, it's essential.

The Cheat Code

One might reasonably ask: How do we know forecasters aren't just identifying spurious correlations themselves? After all, we've seen that correlations can appear predictive without revealing true causation.

What sets good forecasters apart is their consistent performance across diverse domains and time horizons. Unlike the ice cream-drowning correlation that breaks down when conditions change, skilled forecasters maintain accuracy through shifting circumstances. They don't simply identify statistical patterns, they build flexible models accounting for causal mechanisms and context-specific factors.

When forecasters maintain predictive accuracy whether analyzing geopolitics, economics, or public health, they demonstrate something beyond coincidental association. They actively combat the spurious correlation trap through techniques like base rate analysis, scenario testing, and continuous recalibration. Their success persists precisely when simple correlations fail, demonstrating a resilience that validates their methodology as capturing something fundamentally useful about reality.

Indeed, research shows that forecasters with good track records tend to maintain their high performance across various domains, making well-calibrated predictions even outside their areas of expertise. This consistency isn't merely theoretical, it's an empirically verified phenomenon that gives us a powerful tool for assessing world models.

Even more impressive, groups of forecasters consistently outperform individuals. These collaborative efforts combine diverse perspectives and analytical approaches to produce remarkably accurate predictions, often surpassing traditional domain experts. Their strength lies not in specialized knowledge, but in their ability to effectively synthesize expert findings and arguments into coherent, actionable forecasts.

Consider this passage from my fellow forecaster Molly Hickman’s recent article that she wrote in conjunction with Rajashree Agrawal:

The foundational literature (e.g. Goldstein et al, Karvetski et al, Tetlock's Expert Political Judgment) in Tetlockian forecasting seems to point at the top forecasters having better understanding of how the world works than domain experts (if you're on board with the idea that proper scores, in this case Brier, are a reasonable proxy for "understanding of how the world works"). Despite lacking the years of knowledge, experience, and gut-feelings that experts have, forecasters are better at putting numbers to whatever knowledge, experience, and gut-feelings they do have—implying that high-quality expression of shallow information > low-quality expression of deep information. This makes some intuitive sense; for instance, despite farmers' having much deeper understanding of agriculture, it doesn't seem like they're better at crop futures trading than traders.

This observation captures an important insight: the quality of the forecasting process can outweigh the depth of subject-matter expertise. This helps explain why forecasters can succeed across diverse domains: they've optimized the methodology of prediction itself rather than focusing solely on accumulating domain knowledge. The agriculture/trading example illustrates how even in highly specialized fields, good forecasting techniques often triumph over experience-based intuition.

Importantly, these forecasters have the right incentives. They stake their reputations on their predictions, aligning their interests with the pursuit of truth rather than supporting particular agendas or maintaining the status quo.

By tapping into this collective wisdom, we can navigate complex issues more effectively, identify which world models are most accurate in practice, spot emerging trends or potential threats earlier, and make more informed decisions in the face of uncertainty.

As we confront the array of challenges facing civilization, from nuclear risks to climate change, from biosecurity to governance failures, this forecasting approach offers our best path forward. Rather than endless theoretical debates or waiting years for definitive evidence, we can look to these forecasting groups as our reality-checking mechanism and a practical means of determining which world models truly deserve our attention and resources.

In a world of competing narratives and complex threats, good forecasting may be our most valuable compass.

I haven’t had a lot of time for writing lately as teaching, work, forecasting, and life have kept me busy, but I’ve got a few new articles coming soon, including one that’s going out tomorrow morning.

In other news, I’ve been invited to LessOnline this year and I’m looking forward to seeing many of you there. If you’re attending, feel free to say hi or reach out in advance. I’ll be around for the full weekend and always up for a good conversation, especially about forecasting, epistemics, or AI progress.

Finally, Samotsvety is holding a forecasting conference and retreat this June the week after Manifest. While the conference is invite-only, we’re holding a public hangout in Fort Greene Park in Brooklyn on the evening of Tuesday, June 10. If you’re in NYC and interested in forecasting, rationality, or a casual reader of this blog, come by and say hello—everyone’s welcome.

Making a forecast isn’t always the best use of time when making a decision. Most of the time, we’re better off either planning for the worst-case scenario or relying on rules or experts with proven track records. However, there are specific scenarios where investing effort into a structured forecasting process can lead to significantly better outcomes. Let's explore a framework to determine whether to fortify, follow, or forecast when faced with an important decision. By identifying when forecasting truly matters, you can apply it effectively to achieve better results.

Fortify: Prepare for Worst-Case Scenarios

Some decisions are too critical to leave to chance. Take, for example, planning your route for a job interview where traffic could take anywhere from 30 to 60 minutes. Even if you’re 90% confident you can get there in 45 minutes, the stakes are too high to risk being late. In such cases, the precise forecast doesn’t matter, whether the worst-case scenario is a 5% or 50% likelihood, you still plan for it because the stakes are too high. You give yourself 60 minutes to ensure punctuality because the cost of failure is unacceptable. Whether it’s emergency preparedness or ensuring you’re on time for an important event, the goal is to mitigate risks and safeguard against unacceptable outcomes.

Follow: Rely on Rules and Experts

In some domains, experts provide highly reliable forecasts, such as weather predictions. Meteorologists are well-calibrated experts with an empirical track record of accuracy. For example, if you’re planning a beach trip for the weekend and deciding between Saturday and Sunday, it’s practical to trust the weather report to make your choice. Their reliability makes it unnecessary to attempt to outperform them yourself.

Similarly, when the stakes are low and you have personal experience to rely on, following rules that have led to positive outcomes in the past can simplify decisions. For instance, if your favorite restaurant is usually uncrowded on Wednesdays and often features your preferred special, you might choose to dine there based on this reliable pattern. Such rules offer a practical and time-efficient way to navigate everyday choices without overanalyzing.

In both cases, empirical track records guide the decision, offering a reliable foundation for making choices efficiently and effectively.

Forecast: Use Best Practices to Make the Best Decision

Unfortunately, many questions in life are nuanced and involve complicated tradeoffs that don't come with convenient expert solutions. Sometimes the best path forward isn't obvious. Major life decisions, like whether to move to a new city, require personal foresight and deliberate analysis. Follow the most important rule of forecasting and start by gathering base-rate information: What are typical salaries in your field for that city? How happy are people who live there?

For example, if 70% of people in your field who moved to Austin saw their real income increase within two years, that's more informative than your gut feeling about the city's prospects.

Use these data points to anchor your forecast, as base-rate statistics about others’ experiences are often more reliable than your own subjective feelings. People frequently overweight their own predictions about how they’ll feel ("I'll definitely love the weather!"), but trusting what has been true for others is usually a better guide for decision-making.

Why This Matters

Most people are already pretty good at fortifying—planning for worst-case scenarios where the stakes are high (okay, fine, maybe they’re not great with tail risks, but that’s a topic for another post). And unless you’re part of the crowd who thinks wrestling is real, but the moon-landing is fake, you’re probably already mostly following empirically-validated strategies where you should too (another post about how to know which experts to trust is coming soon… HINT: empiricism).

The real opportunity lies in that third category: people rarely take the time to forecast deliberately even when doing so could lead to much better outcomes. Think of the last time you made a major career move - did you look up the base rates for success in similar transitions? Or consider someone choosing between grad school options - how often do they gather systematic data on career outcomes versus just following their gut? By recognizing these opportunities to forecast more carefully, you can make significantly better decisions in areas that really matter.

Forecasting isn't merely about predicting the future; it's a tool for making smarter decisions today by reducing uncertainty and aligning actions with desired outcomes. The next time you face a decision, ask yourself: Should you prepare for the worst case, trust reliable experts, or dive into forecasting? By using this framework and leveraging available data, you can approach uncertainty with clarity and confidence, ultimately improving your results.

Recently, my neighbor Daniel Golliher, author of Maximum New York, proposed a book club. I decided to participate, so I’ve written my reflections on the revised 2020 edition of Robert Putnam's Bowling Alone.

Bowling Alone presents a compelling narrative discussing the decline of social capital in America and supports its case with in-depth analysis of the factors that may have contributed to its decline while endeavoring to compensate for confounding factors and control for spurious correlations.

Social Capital and Its Decline

In Putnam’s framework, social capital refers to the networks of relationships among people who live and work in a society, enabling it to function effectively. Putnam posits that social capital can be broken down into two main categories:

1. Bonding capital refers to connections within homogeneous groups fostering a sense of community and shared identity.

2. Bridging capital refers to connections between heterogeneous groups. The theory proposes that bridging capital can link people to broader opportunities and resources outside their immediate social circles.

Proponents argue that social capital can reduce friction in social interactions and daily life and suggest that higher levels of social capital might correlate with more efficient societies, as trust and established networks could facilitate smoother transactions and collaborations.

As per the emblematic chart above, Bowling Alone argues that social capital in the United States has been declining since the 1950s by presenting decreases in a wide set of indicators of social capital, including civic engagement, political participation, religious involvement, workplace connections, informal social ties, philanthropy, and trust in institutions and other individuals. Putnum provides numerous charts depicting declining participation across a variety of institutions and attributes this to a combination of factors, including increased time spent watching television, two-career families, suburban sprawl, and generational change. He contends that this erosion of social capital has significant implications for community well-being, democratic participation, and overall societal health.

Putnam's approach seems robust. He takes a comprehensive look at the potential causes behind the decline of social capital and provides a well-reasoned rationalization for whether or not he thinks each meaningfully contributed. For example, in evaluating a factor, he might look at trends in comparable countries to determine if the same factor is present there and whether the magnitude of the effect was similar. While Putnam put range estimates on each of the factors he found likely to be responsible as shown in the image below, he left room for unexplained other factors.

Putnam's argument sparked considerable debate and inspired further research into the state of social connections. While his ideas are influential, the mechanisms and applicability of social capital is the subject of ongoing debate. As with many sociological concepts, social capital evades straightforward measurement and quantification.

As a professional forecaster, I found Putnam's approach well-considered and worthy of discussion, and, in my personal judgment, probably directionally accurate. However, with observational evidence, even with extensive controls, it’s hard to draw conclusions with high confidence.

While Putnam's work is thought-provoking, and there are many interesting arguments over the relative importance of the various factors he discussed, I don't want to focus on reanalyzing them since that has been done elsewhere. Instead, I want to look into the unexplained portion of the precipitous declines observed in many of the social and community organizations the book addresses. In the rest of this post, I’d like to explore how network effects might explain an important part of that mystery.

The Power of Connections: Network Effects

Network effects occur when the value or impact of a product or service increases as more people use it. Think of how much more useful a telephone is when the person you want to call also has a telephone. A phone wouldn’t be very useful if you were the only one that had one, or if the only other person who had one was someone you didn’t care much to talk to, but the more people that have them, the more valuable they become. In the context of social capital, we can think of community organizations, civic groups, or even informal social circles as networks whose value grows with increased participation.

One way to quantify network effects is through Metcalfe's Law, which states that the value of a network is proportional to the square of the number of connected users1. Even if the value of a social network doesn’t scale exponentially with the number of users, it seems reasonable to expect that many people would find larger organizations, at least up to a point, to be more valuable since they offer more opportunities to establish both bonding and bridging capital.

Presuming that, for most participants, participating in an organization depends on the cost-benefit balance, in the unfortunate event where a network loses a participant, perhaps due to moving, even if nothing else changes, it could lead to the next marginal member leaving as well as the decreasing value of the network moves their participation from net-positive to net-negative. This could lead to a snowball effect where more and more participants leave. Once enough people stop participating, there is no escaping the vicious cycle.

Combine this concept with rising opportunity-costs such as better at-home entertainment or longer commute times, and we can see a promising explanation for why the decline in organizational participation might have accelerated beyond what Putnam’s factors alone would predict.

This compounding effect could explain why Putnam observed such steep declines in participation across various social organizations. It's not just that people are choosing not to join these groups; it's that the groups themselves become less attractive as they shrink, creating a self-reinforcing cycle of decline.

To better demonstrate this idea, I created a simulation of a community center with a capacity of 200 members where every potential member is randomly assigned a benefit value that they get from the community along with an opportunity-cost. As long as the benefit is greater than the cost, and as long as there is room, the person becomes and stays a member (I also set some other parameters like having the value they get from membership increase slightly over time as they grow more deeply engaged, a random chance of quitting, a maximum monthly join and leave rate, and other variables, but those aren’t important to the overall narrative and you can play with the assumptions if you want to build on my simulation). However, if the cost exceeds the benefit, the member quits.

The simulation starts out with 150 members in 1940. Membership as a percentage of capacity is represented by the blue line, the calculated value of the community (as determined by the network effect, but maxing out in value after reaching saturation at 150 participants) is represented by the orange line. Finally, the baseline societal opportunity-cost is represented by the green line. In my simulation, I added two step-changes in opportunity cost, one after the wide-spread adoption of television and the second after the advent of the internet.

As the simulation starts out, new members rapidly join filling the community cent to its maximum capacity. This remains very stable until the scaled adoption of TV after which some members quit because they find staying home is a better alternative, but they are quickly replaced by other community members who put a higher value on what they get from the community. The network remains relatively stable until the arrival of the internet. For a few years, the value of the network remains high and new members continue to join as other members leave, but as more and more people stay home, the network quickly reaches a point of no return.

Understanding this dynamic is crucial for efforts to rebuild social capital. It suggests that simply addressing individual factors like time constraints or competing entertainment options may not be enough. Instead, strategies to revitalize community engagement might need to focus on reaching a critical mass of participation quickly, leveraging technology to reduce barriers to entry, and creating clear, immediate value propositions for potential members to overcome the inertia of non-participation.

In essence, the power of network effects works both ways – it can amplify growth in social capital when conditions are favorable, but it can also accelerate its decline when various factors align against it. Recognizing this can help us better understand the dramatic shifts in social engagement that Putnam documented and inform more effective approaches to fostering community connections in the future.

It’s easy to underestimate the capabilities of current-generation AI, but with a few adjustments in approach, current models are capable of much more than most people realize.

Better Prompting -> Better Responses

The quality of the prompt significantly affects the result. Providing detailed, step-by-step instructions or better context can dramatically improve results. For example, instructing the model to "think step-by-step" or to "ask clarifying questions" can guide it to produce more accurate and useful outputs.

For example, here’s a prompt I gave to GPT-4o:

Prompt: Jim is Bob's brother. Jim's wife is Sarah who has no siblings. Bob has no other siblings. Sarah's daughter's aunt is named Susan. What is Susan's relationship to Bob?

I gave Claude Sonnet 3.5 the same prompt and it nailed it out of the gate, but this was GPT-4o’s incorrect response:

gpt-4o:

Given the relationships described:

1. Jim is Bob's brother.

2. Jim's wife is Sarah.

3. Sarah has no siblings.

4. …

… the most plausible relationship, without contradicting the given information, is that Susan is Bob's sister-in-law, married to either an undisclosed brother of Jim (and therefore Bob) or related to Sarah through another form of marriage connection.

Yikes!

This response makes me feel kind of sympathetic to the AI is useless crowd, but we can do a lot better. To show what better prompting can do, I created a new session and appended the bolded text below to help GPT-4o get to the right answer on its own:

Prompt: Jim is Bob's brother. Jim's wife is Sarah who has no siblings. Bob has no other siblings. Sarah's daughter's aunt is named Susan. What is Susan's relationship to Bob? Let's think through this logically step-by-step and be careful to make sure you get this right by double checking your answer for inconsistencies with the provided information.

gpt-4o:

To determine Susan's relationship to Bob, let's break down the given relationships step by step:

1. Jim and Bob are brothers.

2. Jim's wife is Sarah, who has no siblings.

3. Bob has no other siblings…

…Given that Susan is identified as an aunt and can only be Bob's wife due to the lack of other siblings and Sarah having no siblings, we conclude:

Susan's relationship to Bob is that Susan is Bob's wife.

This example demonstrates the power of prompt engineering. By providing explicit instructions to think step-by-step and double-check for inconsistencies, we guided the AI to arrive at the correct conclusion. This approach not only improves accuracy but also demonstrates how AI can be steered towards more reliable and logical reasoning.

The takeaway is that this is a skill that can be honed. Just as a manager learns to give clear instructions to their team, users can learn to craft prompts that elicit more accurate and useful responses.

Unfair Comparison: Context and Tuning Matter

Humans often ask models challenging questions they themselves know little about, leading to frustration when the LLM fails on the first attempt. This often results in an unfair comparison, as humans would generally seek additional information or ask clarifying questions before tackling such complex requests. Most consumer-facing LLMs are tuned to provide answers without seeking further context, unlike their human counterparts who instinctively probe for more information or, even more likely, already have sufficient context from past experience specific to the requester. While this kind of single-fire response does well in the context of the chatbot ranking arena, it can lead to frustration in the real world.

For example, look at how two different models respond to a line of questioning that lacks sufficient context.

Prompt: Who is the President?

gemini-1.5-pro-api-0514 : Joe Biden is the current president of the United States.

deepseek-v2-api-0628: As of my last update in October 2023, the President of the United States is Joe Biden. He took office on January 20, 2021.

Prompt: I work for McDonalds. Who is the President?

gemini-1.5-pro-api-0514 : Whether you work at McDonald's or anywhere else doesn't change the answer: the President of the United States is still Joe Biden.

deepseek-v2-api-0628: I work for McDonalds. Who is the President? As of my last update in October 2023, the President of McDonald's Corporation is Chris Kempczinski. He was appointed to this position in November 2019, succeeding Steve Easterbrook.

As the example above illustrates, both models attempt to respond to this ambiguous situation without seeking clarification from the user. The prompt could be interpreted in several ways: Is the user asking about the president of McDonald's, the president of the United States, the president of another country, or even some other president entirely? Instead of asking for more context, the model makes its best guess and proceeds with the response. While it’s technically possible to adapt models to ask clarifying questions at each step, this approach might be less appealing to the average user, who typically expects quick and straightforward answers.

It’s surprising how quickly some users are willing to give up and write these tools off as useless, but I suspect most of them have never experienced the challenges of working with an overseas development team. Although the human-level intelligence of a dev team far surpasses that of any existing LLM, most users would still find it frustrating to accomplish tasks using only asynchronous written communication. For me, the LLM experience, while not the same, is reminiscent of this. When you’re careful to fill in the missing context that most in-person communication takes for granted, you get results.

Overcoming Unreliability

A common objection I hear from the "AI is overhyped" crowd is that AI outputs are unreliable and therefore useless since everything they produce has to be checked by a human anyway. Where this objection falls apart is, in many cases, verification can be automated. For example, in software development, AI can generate code while automated systems verify its correctness. Modern development practices already employ Continuous Integration/Continuous Deployment (CI/CD) pipelines that automatically run unit tests, integration tests, and even performance benchmarks. These same systems can be applied to AI-generated code, ensuring its reliability without human intervention, so developers can focus on higher-level tasks like architecture design and complex problem-solving, rather than writing and testing every line of code manually.

Even when human oversight is necessary, the efficiency gain can still be substantial. AI rapidly produces initial solutions, allowing humans to refine and make high-level decisions rather than starting from scratch, dramatically accelerating the development process. While some may suggest that AI code quality is so buggy that it’s more trouble to use than it’s worth, it’s useful to remember that this is the worst it will ever be going forward and not every company employs a staff of elite programmers like yours does, and in some (most?) cases, the AI is still going to write higher quality code than the human who otherwise would have.

Additionally, using scaffolding techniques, where multiple passes or agents generate a variety of possible answers, can also enhance the output quality. This method allows a terminal model (or potentially even you) to select the best result from a set of responses, leveraging the model’s ability to generate diverse solutions. Incorporating a variety of models into the solutioning and refinement pipeline improves outputs even more.

As an example, here’s an interaction with GPT-4o mini applying just the simple action of performing multiple passes to allow this small model to self-correct (I had to use mini because GPT-4o and Claud kept getting my questions right on the first try).

Prompt: please give me five sentences with exactly five words each and number them 1 - 5.

gpt-4o-mini:
1. The cat chased the mouse.
2. She enjoys reading mystery novels.
3. He runs every morning before breakfast.
4. They traveled to Paris last summer.
5. Coffee helps me stay awake.

Prompt: which of those sentences actually have 5 words? Please just output the numbers.

gpt-4o-mini: 1, 2, 3, 4, 5

Prompt: which of those sentences actually have exactly 5 words and no more? Please count the words from each one and display it after the sentence.

gpt-4o-mini:
1. The cat chased the mouse. (5)
2. She enjoys reading mystery novels. (5)
3. He runs every morning before breakfast. (6)
4. They traveled to Paris last summer. (6)
5. Coffee helps me stay awake. (5)

Prompt: now tell me which actually have 5 words please.

gpt-4o-mini: 1, 2, 5

In essence, the "unreliability" argument underestimates both the power of automated verification and the value of AI as a force multiplier for human productivity. As AI systems continue to improve, the balance of effort will increasingly shift from verification to strategic application of AI-generated outputs, leading to significant efficiency gains.

Conclusion

By enhancing our interactions with large language models—through better prompting, scaffolding, and feedback—we unlock their true potential. Feedback is critical for improving model performance. Unlike humans, who naturally validate their answers against real-world feedback, deployed models have to rely on explicit feedback mechanisms. For instance, when writing a technical report, a human might cross-check data against reliable sources to ensure accuracy. Implementing similar feedback loops for models can help refine their outputs. This approach allows us to appreciate the capabilities of AI and the efficiency gains achievable when humans and AI work together effectively.

Forecasting performance evaluation is muddled by the convolution of skill and luck. In a realm where numerous forecasters make predictions over a large sample of events, a critical question emerges: how often are the top performers genuinely skilled as opposed to just lucky?

Track Record Limitations

While a forecaster’s track record is a useful indicator, there are limits on the conclusions we can reliably draw. While it’s true that, over a long enough time span, a forecaster’s improper calibration will catch up with them, all but the most prolific forecasters have at most dozens of resolved questions. So when can we be confident that a forecaster is actually adding signal rather than noise?

Bernoulli Trials in Forecast Evaluation

To evaluate whether a forecaster is skilled or merely lucky, we can use the concept of Bernoulli trials. In a Bernoulli trial, there are only two possible outcomes: success or failure. By treating each forecast as a series of Bernoulli trials, we can apply statistical hypothesis testing to assess the likelihood that the observed performance was due to luck rather than skill.

Imagine a forecaster who, time and again, predicts a 100% chance for events that, in reality, have a 95% probability of occurring and where the crowd forecasts the ground truth probability accurately. Further suppose that they have a perfect track record over a history of 40 resolved questions. At first glance, their record of success might seem impressive. However, this could be a classic case of picking up pennies in front of a steamroller—reaping small gains in the short term while ignoring the looming risk of a catastrophic loss. At the outset, we know the most likely outcome would be to observe 2 negative resolutions out of the 40 questions, but what are the chances that all 40 would resolve positively?

Hypothesis Testing

We set up our hypothesis test with the following:

• Null Hypothesis (H0): The forecaster's success is due to luck (true probability of success for each event is 95%, as estimated by the crowd).

• Alternative Hypothesis (H1): The forecaster's success is due to skill (probability of success differs from 95% for each event).

Using the binomial distribution, which is appropriate for a series of independent Bernoulli trials, we calculate the probability of observing the forecaster's success rate under the null hypothesis.

A Lucky Streak?

Imagine a forecaster predicts a 100% chance of success for 20 events, and all 20 events occur. At first glance, their forecasting ability seems remarkable. However, our hypothesis test reveals a different story. With a P-value of approximately 0.358, we find there's a 36% chance of such a success rate occurring by luck alone, given the crowd's 95% estimation. This high P-value suggests we shouldn’t confidently attribute the forecaster's success to skill.

The problem is compounded by the sheer number of people forecasting. With so many forecasters, some will inevitably get lucky. Moreover, there are many forecasters who are at least somewhat talented and happen to ride a wave of luck on the questions they're wrong about. This mix of partial skill and good fortune can create the illusion of consistent proficiency.

Another critical factor is the issue of correlated questions. This can be particularly pernicious in scenarios like election forecasting. For instance, if a forecaster is overconfident in a particular party and happens to be correct by luck, they will likely outperform the crowd across most questions in that category. This correlation can significantly skew the perceived accuracy and skill of the forecaster, as their success is not independent across different questions.

Additionally, polling error often tends to be skewed in a particular direction due to systemic biases, methodological issues, or unforeseen events. A forecaster who aligns with the direction of this skew can appear highly accurate purely due to this bias. When the polling error benefits a particular prediction, it can reinforce the forecaster's apparent success, further complicating the evaluation of their true skill.

Conclusion

This underscores the importance of prudence in forecaster evaluation. Relying on statistical methods like hypothesis testing provides a more grounded assessment of a forecaster's ability. It reminds us that what might look like skill could simply be luck, or some combination of luck and skill. It also reminds us why it’s important to look at crowd medians instead of relying too much on any single forecaster. Aggregated forecasts leverage the wisdom of the crowd, reducing the impact of any single person's lucky streak and providing a more reliable and balanced perspective.

Disagreements are an excellent opportunity for improving our models of the world, but expressing dissent constructively can be challenging. In my experience, people often don't have well-formulated ideas about the underlying probability of the point they're arguing for, but instead start from a position that confirms their worldview. In a conversational disagreement, there is little incentive for them to concede ground since often they're arguing for a point that isn't even falsifiable. Introducing stakes, as with monetary bets, can help participants critically evaluate their beliefs and the operationalization required to formulate the bet can be clarifying. However, challenging someone to engage in a monetary bet can feel confrontational and inappropriate in many social settings. As an alternative to monetary bets, I propose the concept of 'book bets' as a more congenial alternative.

Book Bets in Concept

In the same way that monetary bets allow participants to express confidence in their beliefs, book bets provide a mechanism to put their intellectual “money” where their mouth is by wagering the number of book suggestions from the other party they’re willing to risk reading if they’re wrong. The odds of the bet can reflect the confidence each person has in their position.

Here's how it works:

• Two people disagree on a forecast

• They define a specific, falsifiable claim, a timeframe, and a trusted source for resolution

• They agree on the odds of the bet; any odds ratio between the differing beliefs should be acceptable, but this is an opportunity for the more confident party to be magnanimous and accept odds closer to their stated estimate as proof of their confidence

• After the forecast is resolved, the correct participant gets to select the specified number of books for the other participant to read

By using books to read as the stakes, ‘book bets’ create a friendly atmosphere, fostering intellectual growth and encouraging constructive dialogue. Hopefully, the winner will select books that will provide insights into the perspectives that helped the them succeed in the bet and help the other participant better understand their perspective.

Book Bets in Practice

Suppose two co-workers, Mark and Dana, are having lunch together and the subject of the future of self-driving cars comes up. Mark, who has recently seen some exciting announcements on social media excitedly claims, "Self-driving cars are going to be everywhere very soon!" Dana, skeptical, responds, "hmmm, I’m not so sure. It feels like there are still a lot of things that need to fall into place before that can happen.”
Mark replies, “I’m usually right about these kinds of things. I follow very reliable sources and I’m telling you, it’s happening.”

Seeing an opportunity to turn their debate into something more constructive, Dana proposes, "How about we make this interesting with a book bet? Let's define what 'very soon' means and see who's closer to the mark when the time comes."

Mark considers this, then moderates his initial statement, "Okay, let's say by the end of next year, autonomous taxis will be generally available in the same way as ride-share is to virtually anyone in every major city in the US. The technology is practically solved. I’d give it a 95% chance."

Dana nods, “If you’re that confident, how about if you agree to read 19 of my book suggestions if you’re wrong and I can read 1 of your book suggestions if you’re right? Those odds would match your 95% estimate.”

Mark thinks for a second about having to read 19 books if he’s wrong and considers whether he’s really so sure, “Well, no. I should have said that even though I think the technology is ready, a lot of cities might have red tape that could take a while longer to get sorted out, so maybe I should have said at least 6 of the 10 biggest cities in the US will have self-driving taxis by the end of next year.”

Dana nods, "That seems more plausible than 'everywhere very soon,' would you like to make the bet at 19 books to 1 with that caveat in place?”

Mark thinks for another second or two, “hmmm, I guess the regulatory stuff adds some uncertainty, but, I’m still pretty sure. Probably 80%. How about you Dana? What do you think?”

Dana replies, “I haven’t really thought about it, but if you’d asked me this morning, I probably would have said only 15%, but there have been a lot of times in meetings that you spotted something everyone else missed, and you read about these kinds of things a lot more than me, so I guess I’d update to around 25%. I’m really not very confident though. All I can really say is I’d be kind of surprised if it turns out to be true, but, if it does, I’d be happy to read a book of your choosing. Since you’re more confident, how about if we set the odds at 2:1. Would you be willing to read two of my suggestions if you turn out to be wrong?”

Mark says, “It’s a deal!”

Reflections and Suggestions

Book bets are a practical tool that turns casual debates into serious discussions with real stakes. They impose a cost on overconfidence without being aggressive or adversarial and encourage honest belief representation. When participants risk having to immerse themselves in readings that challenge their views, they’re more likely to measure their words and think critically about their assertions.

• Real Stakes Mean Real Consideration: The introduction of real stakes—having to commit time to reading books—compels participants to evaluate their certainty more cautiously. While most people feel good making confident assertions that support their existing worldview when there are no costs to face, once their beliefs become tethered to real-world consequences, you can sometimes see an immediate shift as their rational mind jumps in to save them from the potential cost of overconfidence.

• Turning Arguments into Resolvable Questions: Like monetary bets, book bets can cut through the noise of stubborn arguments by focusing on verifiable outcomes. They move theoretical disagreements into concrete, measurable terms, which is a step towards uncovering truth.

• Opportunity for Learning: The beauty lies in the aftermath for the non-winner. The selected readings offer a window into the winner's thought process, potentially shedding light on why they ended up on the right side of maybe. This isn’t about conceding defeat; it’s seeing a new perspective that could reshape their thinking.

Conclusion

Book bets offer an alternative to traditional bets, moving potential intellectual conflicts into opportunities for growth and understanding. By swapping monetary stakes with the promise of reading, these bets lower the social barriers to entry for more people to engage in meaningful, evidence-based discussions. It’s not about proving who’s right, but about better understanding the world and each other.

Next time you find yourself in a spirited debate, consider proposing a book bet. It could turn a disagreement into a learning opportunity.

Last year, Scott Alexander reached out to ask Samotsvety to participate in the ACX Forecasting Contest to provide a benchmark for what a “top” forecasting team could do. I made forecasts on behalf of the group, but they were reviewed by several members to make sure there weren’t obvious mistakes. I’ve put together a few thoughts on the experience.

Contest Results

When the contest was scored, Samotsvety secured a position in the 98th percentile. This outcome not only surpassed the median superforecaster, who landed in the 70th percentile but also outperformed Manifold, with its 89th percentile score and even edged out the superforecaster aggregate which landed at the 97.5th percentile. Meanwhile, perhaps unsurprisingly, the Metaculus algorithm ended up in the 99.5th percentile. Although the rankings aligned with my expectations for overall capabilities, I was a little surprised to see so many well-calibrated contenders in the upper echelons of the contest results.

Subscribe now

Wait… What?

Yeah, you heard me. I’ve previously expressed concerns about forecasting contests, and, for reasons that should be clear from that article, it’s not that uncommon to see some of the top slots in forecasting tournaments go to forecasters with decidedly unimpressive long-run track records. I was a little surprised to have done so well because I expected more participants to have “gone for broke” in order to win since there were prizes at stake and there were clusters of questions that I would have expected to be pretty correlated (e.g. Trump / Desantis).

It Happened Before

The go-for-broke strategy has actually happened in previous ACX forecast contests. For example, Zach Stein-Perlman, a top contender in the 2022 ACX contest wrote:

My expected score is slightly worse than it would be if I always gave my true probabilities. I mention this in case you want to exclude me from analysis for that reason. (The form says "there is no strategic advantage to putting anything other than your honest predictions for each event", but this is totally false: I don't care about expected score, just probability of doing very well. If Scott had said to be honest, I would have, but instead he said "The winner will get eternal glory" so I'm lazily* maximizing winning probability. *If the universe was at stake, I would consider other tactics, but it's not, so I'm just being overconfident.

Zach is absolutely right: within the confines of a single tournament, it usually pays to be overconfident within clusters of correlated questions, but selecting the optimal one-off strategy takes some finesse. Going for broke across too many question clusters increases the chances of having your overconfidence catch up with you.

It Probably Would Have Worked

As an example of how this strategy could have worked, look at questions 34 - 36 which I would consider as a correlated question cluster.

34. Will Bitcoin go up over 2023?

35. Will Bitcoin end 2023 above $30,000?

36. Will Tether de-peg in 2023?

Since Samotsvety has a long-run reputation to maintain, I gave my true probabilities when I made my forecasts. I deferred to a Montecarlo simulation which suggested that Bitcoin was relatively unlikely (~34%) to end the year above $30,000 although this was less confident than the Metaculus median which was even lower, so this helped my overall performance in the tournament. Meanwhile, if a participant wanted to “go for broke” on this cluster of questions, they might start by assuming that 35 would be true which would imply that 34 was also true and would suggest that Tether would be unlikely to de-peg. In hindsight, we know that this would have paid off, but, at the time I entered my estimates, I would have estimated this strategy to have a ~30% chance of paying off (i.e. 34% for Bitcoin going above $30K times ~88% that Tether wouldn’t de-peg given Bitcoin’s rally times a 100% chance that bitcoin would go up). Of course this strategy comes along with a ~70% downside chance of absolute humiliation for making absurdly overconfident forecasts, but, for some, maybe this is a risk worth taking for a chance at “eternal glory”. So if the go-for-broke participant had paired this with a strategy of extremizing anything within a 15% chance of an expected outcome according to the Metaculus medians (~41% chance of success1 assuming the medians were reasonably well-calibrated, but we know with hindsight this would have worked for 2023) while using the Metaculus medians for the rest, it would have had about a 12% overall chance of working (30% * 41%) and probably would have beaten the Metaculus algorithm placing the participant within striking distance of winning the entire tournament, especially if they made a few other confident forecasts. While it’s true that any single go-for-broke participant is unlikely to select the right question cluster(s), across more than 3000 participants, I would have expected at least some to stumble on to the right ones.

Why Didn’t Go-For-Broke Win This Year?

I expect some contestants did employ the go-for-broke strategy, but, as I mentioned before, it requires finesse, and, in a contest with this many relatively distinct correlated question clusters, I expect anyone who pulls it off to also be a pretty good forecaster. Of course, in some cases, extreme overconfidence looks the same as going for broke, but, when trying to optimize to win a tournament, it helps to have a more deliberate strategy. Here are my theories about why the day seems to have been carried by well-calibrated forecasters.

Go-For-Broke Participants Got Greedy

When employing the go-for-broke strategy, it’s easy to get greedy. Take the implications about Bitcoin’s assumed price increase and apply them to questions 30 - 33.

30. Will US CPI inflation for 2023 average above 4% in 2023?

31. Will the S&P 500 index go up over 2023?

32. Will the S&P 500 index reach a new all-time high in 2023?

33. Will the Shanghai index of Chinese stocks go up over 2023?

Operating as if we knew that Bitcoin would end 2023 at $30,000 would suggest that the interest rates probably stayed relatively low. Maybe they even went down. In either case, if we think interest rates never rose too much so as to prevent Bitcoin from rallying, it indicates inflation probably came down from its highs (maybe a 70% chance). That leads us to update on question 30 (CPI inflation). It seems more unlikely since we suspect interest rates never went that high. Let’s go for broke and put question 30 as 0%. What about questions 31 (S&P goes up) and 32 (S&P new high)? If we think CPI came in low and interest rates stayed down, the S&P 500 probably went up (85%) and might have even hit an all-time high (maybe still only a 55% chance given that we were down so much earlier in the year). While question 31 came within a hair’s breadth of happening, extremizing on this question would have tanked the game. Now add insult to injury with question 33 (Shanghai index goes up) which we might have also assumed to be more likely to have gone up and thus extremized, but, in hindsight, we know that would have gone very badly.

Go-For-Broke Participants Picked the Wrong Clusters

As per the previous section, if we had started with the question cluster including questions 30-33, there wouldn’t have been a successful way to convert that into a winning extremized narrative for 2023. Starting with the most unlikely outcome that the S&P 500 would hit an all time high would have been ruinous in 2023 even though it came very close to happening and would have led to many other correct conclusions. With log scores, overconfidence leads to eternal damnation.

Go-For-Broke Picked the Wrong Side Within a Cluster

Picking the wrong side of a bet is ruinous when going for broke and within correlated clusters, there are cascading implications. For example, questions 17 and 18 were:

17. At the end of 2023, will prediction markets say Donald Trump is the most likely Republican nominee for President in 2024?

18. At the end of 2023, will prediction markets say Ron DeSantis is the most likely Republican nominee for President in 2024?

It’s possible that both could have resolved as false and an entirely different candidate could have taken the lead, but if either one of these had resolved as true it would imply the other would have to resolve as false. If someone had gone all in on Desantis it would tanked them both on the Desantis and the Trump questions.

Similarly if someone had wanted to go all in on Gavin Newsom, they would have lost both on that and on the Biden question and we now know it wouldn’t have worked out in this universe.

The Tournament Was Well-Designed

Even though there were many correlated question clusters, there were enough distinct / loosely tied clusters to make it hard enough to game the system this year. When I see absurdly overconfident forecasters win tournaments it’s usually when the contest is limited to a few highly correlated questions (e.g. an elections forecasting tournament).

Most People Wanted to Do Their Best “Honestly”

I should make it clear that although we’ve never met, I have a lot of respect for Zach (the 2022 high-performing contender mentioned above) and I don’t think it’s unethical to go-for-broke as part of having fun in an online contest, especially if you make it clear that’s what you’re doing, but I would guess the majority of participants just wanted to play according to the intention of the contest and see how they would stack up.

Final Thoughts

I’m pretty confident the Metaculus algorithm came up with objectively better-calibrated forecasts than me and that many of the other leading participants, like Peter Wildeford2 and Ezra Karger3, really did employ robust strategies that were not just luck or overconfidence. Although there were probably a few lucky forecasters in the top ranks this time, I think many of the contest’s top contenders will continue to be top performers in the years to come. On the other hand, it’s almost certain that Samotsvety would have done better if we had all contributed independent forecasts and then aggregated them as per our usual method, but, unfortunately, the group just didn’t have the bandwidth. The group was able to steer me away from at least one clearly suboptimal forecast, but with the benefit of hindsight there are definitely a few I could have done better on. If I get the time, I’d like to write up a formal post-mortem on a few of these, but I have so many half-completed drafts in my queue that I probably won’t get around to it until 2027, and, by that time, I’ll be able to just have an AI do it for me. For that matter, I kind of expect AIs to be better forecasters than me for most questions by then too, but that’s a topic for another time. Although I had a lot of fun thinking about the questions for 2023, due to personal time constraints, I didn’t get a chance to participate for 2024, but I’d like to deep dive on select questions if I get a chance. Meanwhile, Jack4 made some very good forecasts for this year’s contest which I mostly agreed with. If you want to peer ahead into the future, I’d suggest starting there.

Subscribe now