Judicious Prospecting

Finding opportunities worth pursuing

Recently, a friend sent me an article on “really trying” that outlined a series of anecdotes (not intended in the pejorative sense of how the word is frequently used) about how belief in success can lead to the inspiration and perseverance to achieve what others found intractable. It’s good, and you should read it if you have time, but the gist is that a lot of stagnation could be avoided if people just kept up a high level of commitment to finding solutions and continuously challenged themselves to keep pushing.

Here is an excerpt from the article:

Baker and Minkyung Baek, a postdoctoral fellow in his lab, saw an opportunity. They might not have the code that the DeepMind team used to solve the protein structure problem, but they knew it could be done.

The article suggests the knowledge that the problem had a solution was enough to motivate the lab to discover it.

I saw something similar happen recently with the union-closed sets conjecture: for a long time there had been little progress, but then, someone made an incremental, yet impressive, advance and, within days, more advances started happening. Even I started playing with it.

My interpretation of the mechanism driving the sudden surge in breakthroughs is different though. While the article is an important reminder that, sometimes, all you need is belief and tenacity to accomplish the task at hand, there is also another perspective worth exploring.

The Parable of the Judicious Prospector

When entering an unexplored area, the judicious prospector looks over the terrain to see if there are treasures lying out for easy taking. They might examine the rock outcrops for signs of exposed mineral veins, but, if a quick check doesn’t reveal anything, they move along.

All the while, other prospectors are digging for treasure in various places for all different reasons. Some of them might have plausible theories like “quartz deposits can be an indicator that gold is near by, so dig there”. Others might use the equivalent of dowsing rods and occasionally just get a lucky break (or more likely not).

Most of the prospectors waste time and energy digging in fruitless areas hoping for a lucky break that never materializes, but, when one of them strikes gold, it sends a strong signal to the other prospectors: dig here!

Not having wasted energy and resources on fruitless efforts, the judicious prospector is ready to seize the best opportunities as they arise.

My Interpretation

It actually makes sense that, even when the expected value is high, most people aren’t going to put a lot of effort into a project where the most likely outcome is failure, and, in most cases, it’s probably the right decision.

For financial investments, most investors are willing to forego higher returns for a portion of their portfolio in exchange for safer assets (lower volatility), so it makes sense that few people would be willing to go “all in” on uncertain projects. The more uncertain the project, the higher the expected value of the reward needs to be.

The announcement of a new discovery in a long dry area changes the expected value and provides a signal that this might be an area worth spending resources investigating. The information that someone found a solution indicates that progress on the problem can be made without going too much deeper than where others have already explored.

The “judicious prospecting” mindset is worth balancing against the “really trying” mindset. There are times where both frameworks are useful, but distinguishing one from the other is a challenge. As a possible solution, a friend suggested that organizations could use prediction markets to evaluate grant proposals. Taking it a step further, there are a multitude of scenarios where futarchy could shed light on where to best apply effort. It’s an interesting concept, and it has intriguing possibilities, but we’ll leave further exploration for another post.