One more newsletter for you before we publish our final FLIPR forecast shortly after midnight.
asserted
we → publish → midnight
If you’re new here, we do not usually post this much: it’s typically ~3 items a week, not three a day.
asserted
it → ’re → much
But we hope you’ll join us for our final flurry of coverage, including our Election Night livestream and live blog tomorrow.
asserted
you → hope → livestream
And we have a bonus episode of Still Counting out:
asserted
we → have → episode
Reviews, comments, likes and subscribes are always greatly appreciated.
asserted
Reviews → appreciate → ?
We covered a lot of ground on the show.
asserted
We → cover → show
But I wrote about those NYT polls and Haley Stevens yesterday, so I’m going to use this as an excuse to cover another topic that Galen, Clare and I were only able to scratch the surface of.
asserted
Galen → write → surface
I thought it would be nice to write about this ahead of time so it doesn’t feel like spin if a lot of FLIPR’s forecasts don’t come true.
asserted
lot → think → forecasts
I’ll sometimes get the question, “How do you know if a probabilistic forecast is ever really wrong?” which is an entirely reasonable thing to ask.
asserted
which → get → question
In fact, let me give you a four-part answer.
asserted
me → let → answer
It’s certainly not quite as simple as saying “30 percent chances are supposed to happen 30 percent of the time.
asserted
chances → ’ → time
That’s true insofar as it goes, but it’s incomplete.
asserted
it → ’ → ?
And if you’re being Bayesian about it, you can’t always use this as a get-of-of-jail-free card.
asserted
you → use → card
A 0.03 percent chance happening is much more suspect for your model than a 30 percent hitting.
asserted
chance → happen → hitting
You want your probabilistic forecasts to be well-calibrated over a sufficiently large sample of true out-of-sample predictions.
asserted
forecasts → want → predictions
In other words, if you’ve forecasted 100 events that you said had a 30 percent probability of happening, and 27 of them or 34 of them actually did “hit”
asserted
27 → forecast → them
, your model is doing well.
asserted
model → do → ?
Backtesting doesn’t count.
asserted
Backtesting → count → ?
Any halfway-competently-designed model should fit the past data well; that’s the easy part.
asserted
that → design → data
But the model may be misfit or overfit, keying off of essentially random factoids in past data that have no true predictive power and aren’t likely to repeat themselves.
uncertain
that → key → themselves
You may have noticed that in documenting our models, we tend to not make claims like “ELWAY would have gotten the exact score right in the Super Bowl XVI” or “FLIPR would have nailed the 1994 midterm.”
uncertain
FLIPR → notice → midterm
Who cares.
asserted
Who → care → ?
We already know what happened.
asserted
what → know → ?
I generally think it’s a negative sign when forecasters brag about their backtested results instead of explaining why they think their models make structural sense.
asserted
models → think → sense
In the case of the Silver Bulletin/FiveThirtyEight lineage of models, however, we have something fairly unique: an 18-year-track record of making true out-of-sample predictions in advance of elections and sporting events, which were published ahead of time, and indeed have received a ton of public scrutiny.
asserted
which → have → scrutiny
At FiveThirtyEight, we actually had an interactive that diligently tracked our forecasts (although the Internet Archive version of the interactive is a little hinky and Disney nuked the original version).
asserted
Disney → have → version
Our sports forecasts were extremely well-calibrated: over a large sample size, the 30 percent chances really did hit about 30 percent of the time, and so on.
asserted
chances → calibrate → time
Our politics forecasts were pretty well-calibrated too, although actually a bit underconfident: there were actually too few upsets relative to what our models expected.
asserted
models → calibrate → what
It’s hard to know whether this is a statistically significant differential1, but it makes me a bit less worried when FLIPR is more confident about a race than the consensus.
asserted
FLIPR → ’ → consensus
A second way is a very Nate way that some of you won’t care for.
asserted
some → care → you
Would the model have made you money if you’d been betting on it — relative to other models, the conventional wisdom as far as it can be sussed out, or betting markets?
asserted
it → make → models
If you trusted our model in 2016, you should actually have made a large bet on Trump, because you could get roughly 5-1 or 6-1 odds when our model thought he’d win around 1 time in 3 instead.
uncertain
he → trust → 3
We could get into a more detailed discussion about things like Brier scores — you not only want your model to be well-calibrated, you also want it to discriminate clearly between relatively more and relatively less likely outcomes.
uncertain
it → get → outcomes
You’ll be calibrated perfectly if you always forecast every race as 50/50 but the model doesn’t tell you anything.
asserted
model → calibrate → anything
But I actually think some of the most important ways to evaluate a model are qualitative.
asserted
some → think → model
The third thing I always look at is whether a model responds proportionally to new evidence.
asserted
model → look → evidence
If either the magnitude or certainly the direction of changes in our model doesn’t pass a common-sense test in accordance with the new evidence, that’s often a sign it’s poorly constructed.
asserted
it → pass → evidence
For instance, when that set of very strong NYT/Siena polls for Democrats came out last night, their win probability in the Senate improved by about 2 percentage points in FLIPR.
asserted
probability → come → FLIPR
That seems fairly rational.
asserted
That → seem → ?
And the NYT/Siena polls have generally been on the higher side of the averages for Dems, so FLIPR was expecting some relatively good numbers.
asserted
FLIPR → expect → numbers
…and 48 more, not listed.
The Aporia
