I'm really interested in your thoughts on when it makes sense to use log scoring! It occurs to me that one striking illustration of the problem with mean absolute error is that if something happens 50% of the time, your MAE will be the same no matter what your forecast is. If you said that coin flips never come up heads, your MAE will be the same as it would be if you correctly predicted they'd up heads half the time.
You're right on with your assessment of MAE. Meanwhile, I actually took out the sections on log scoring, Brier scoring, and betting markets to put in a different post because this one was too long for email (you know that dreaded blue bar warning you get from substack when you feel like your writing is finally starting to flow). My bottom line with Brier vs log scoring is that, if the directionality of the mistake doesn't matter (e.g. if the ground truth is 1% and estimating 0% or 2% are equally costly), Brier scoring is the better choice since being off by the same amount in either direction will yield the same expected error over the long run. Certainly we could contrive an example to fit this loss function, but I think in most cases in the real world directionality does matter in the sense that it is far more costly to be overconfident than under confident as we approach certainty and log scoring is the better choice for those circumstances (with infinite penalties for complete confidence that turns out to be incorrect). I feel like log scoring is also more intuitive in this sense since it maps more closely to betting markets.
I'm really interested in your thoughts on when it makes sense to use log scoring! It occurs to me that one striking illustration of the problem with mean absolute error is that if something happens 50% of the time, your MAE will be the same no matter what your forecast is. If you said that coin flips never come up heads, your MAE will be the same as it would be if you correctly predicted they'd up heads half the time.
You're right on with your assessment of MAE. Meanwhile, I actually took out the sections on log scoring, Brier scoring, and betting markets to put in a different post because this one was too long for email (you know that dreaded blue bar warning you get from substack when you feel like your writing is finally starting to flow). My bottom line with Brier vs log scoring is that, if the directionality of the mistake doesn't matter (e.g. if the ground truth is 1% and estimating 0% or 2% are equally costly), Brier scoring is the better choice since being off by the same amount in either direction will yield the same expected error over the long run. Certainly we could contrive an example to fit this loss function, but I think in most cases in the real world directionality does matter in the sense that it is far more costly to be overconfident than under confident as we approach certainty and log scoring is the better choice for those circumstances (with infinite penalties for complete confidence that turns out to be incorrect). I feel like log scoring is also more intuitive in this sense since it maps more closely to betting markets.