How wrong we are, and where.

This is the scoreboard. It covers one mountain, one quiet spring, and a small fraction of the forecasts we issued. Read the first table before the headline number — the headline is carried almost entirely by the hours when nothing was happening.

When we said it would be busy, we were wrong

Forecasts are grouped by what the model itself said the crowd level would be, then scored against what the cameras and skier reports saw. This is the table we would want to read about someone else's forecast, so it goes first.

The pooled number, in context

Scored against camera-derived waits at Mammoth, the day-of forecast has a mean absolute error of 4.18 minutes across 554 forecast-hours over 100 lift-days at Mammoth, April 5 to June 10, 2026. Over those same hours the average line was 2.11 minutes long, so most of that error is us over-calling a quiet lift. It covers 12.7 percent of the forecasts issued in that window; the rest were never observed and so were never scored.

The window, and the number without it

A lift-hour is scored only when (a) every camera observation behind it was produced by the wait conversion currently in production, and (b) it falls inside the period of continuous scheduled collection.

The published figure has to answer one question: how well does the system that exists today forecast? Observations produced by a conversion that has since been deleted cannot answer it, and neither can days on which the collector was not running.

The window removes 125 scored hours, taking the record from 679 hours to 554. This is every hour it removes, and the rule that removes it.

2026-03-19 to 2026-03-31

29 scored hours. Ground truth produced by the pre-Q/mu conversion; only two of the five scored cameras existed; no scheduler. This is the build period. This block is the single largest mover of the headline. It is removed on the conversion-change rule, which was fixed before the per-month figures were looked at.

2026-04-01 to 2026-04-04

39 scored hours. Ground truth still produced by the M/D/1 formula (deleted 2026-04-04 12:11) or by the pre-model path, and the camera set was still changing. Removing these four days makes the headline WORSE than a plain 'drop March' cut would (4.18 against 4.25). They are removed because the rule says so.

after 2026-06-10

57 scored hours. Outside continuous collection — the scheduler had been off for two months. These are off-season dev runs.

Quoting 4.18 minutes on its own would be technically true and substantively misleading. Against a line that averaged 2.11 minutes, an error of 4.18 minutes is not precision — it is us calling a crowd that was not there.

Both series, including the one that is not a forecast

How little of this gets scored

A forecast-hour can be scored only where a camera or a skier saw that lift in that hour. In the published window we issued 4,346 lift-hours of forecast and could check 554 of them — 12.7 percent. Across the whole record 6 lift lines at one resort carry a camera; 5 of them recorded frames inside this window, and 9 lifts were scored — because a lift can be scored from skier reports with no camera on it at all. Every other lift in the app is a forecast nobody has ever checked.

Per lift

Only five lifts have enough scored hours to say anything about.

Four more lifts — Gold Rush Express, Stump Alley Express, Roller Coaster Express, Chair 23 — have between 1 and 2 scored hours each. That is not a sample, so we are not printing a number next to them.

Is the forecast wrong, or is the thing measuring it wrong?

Ground truth is not measured. Cameras count heads; a queue model divides by an asserted service rate to get minutes. If that service rate is too generous, the derived 'actual' is too low, and a correct forecast would look like constant over-prediction. The observed bias is +3.96 and remarkably one-directional, which is the signature a biased ruler would leave.

Hours the ruler cannot be blamed for

Isolate in-window hours in which EVERY camera frame saw 10 or fewer people in the queue. On those hours the derived wait is 0 because of the zero-threshold rule, not because of a division — no service-rate error can change them. Compare the forecast's bias there against its bias everywhere else.

This is a bound, not a measurement. It rests on the counterfactual being computable from stored columns (it is: the conversion reproduces from live_crowd.queue_filtered and live_crowd.service_rate on 2,588 of 2,591 in-window frames, the three misses being 1-minute rounding). It cannot be closed without independently timed waits.

The same forecast, in the unit we actually measure

A camera counts heads. Minutes are inferred from heads by a queue model. So here is the same forecast scored in heads, using each lift-hour's own measured service rate.

What is wrong with our forecast

First person, no hedging, in the order that matters.

1. The forecast runs high, almost always. Bias is +3.96 minutes against a mean observed wait of 2.11 minutes. 95 percent of the average error is one-directional over-prediction, not scatter.
2. When the forecast says the line is busy, it is usually wrong. On hours the model labeled 'high', it predicted 19.9 minutes and the line ran 2.4. That is a 17.5 minute average error on exactly the hours a rider would act on.
3. The number is built on a quiet spring. Every scored hour is from 5 April to 3 June at Mammoth. The mean line was 2.1 minutes long. Nothing here says anything about a Saturday in January.
4. Almost nothing gets scored. 554 of 4,346 issued forecast-hours had any ground truth at all, which is 12.7 percent. Five lift lines at one resort have cameras. Every other lift in the app is unverified, including all 51 lifts forecast at the seven other resorts.
5. The ground truth is itself a model output. A camera counts heads; a queue model turns heads into minutes using service rates that were asserted, not measured. The best-fitting correction to those rates is about 1.5x and accounts for roughly a third of the over-prediction. It does not account for the rest.
6. Any line of ten or fewer people is recorded as a zero-minute wait by rule. That is 29 percent of every camera frame inside the window.
7. The live corrector barely corrects. On the 84 hours where both series exist it changed 12 of them and improved the average error by 0.42 minutes, and it did that by reading the same observations it is then scored against.
8. Twelve and a half percent of the scored baseline hours come from prediction rows written after the hour they describe. Those are hindcasts sitting inside a forecast number.
9. The published figure is a windowed figure. 125 scored hours were excluded by the window rule, including the 29 worst hours on record. The unwindowed number is 5.34 and is published beside it.

What the ground truth actually is

Our ground truth is 3,635 camera observations across 5 lift lines at mammoth (2026-04-05 to 2026-06-07), plus 13 skier-submitted waits across 7 lifts. It is not a stopwatch.

• The median frame saw 15 people in line; the largest saw 160.
• A line of ten or fewer people is recorded as a zero-minute wait by rule. That is 29.2 percent of every frame we have.
• No stopwatch has ever been held against this model. Not once. The conversion from heads to minutes has never been checked against a timed wait.

Numbers we withdrew

These were published and are not true as stated. They are listed rather than deleted, because a correction that leaves no trace is not a correction.

Scope of everything above

Resorts

mammoth — Mammoth is the only resort with any ground truth. Every other resort in the catalog has predictions and zero observations.

Window

2026-04-05 to 2026-06-10, 29 days with predictions

Excluded

resort_slug = 'simulator' — Synthetic data generated by the mountain sim on 2026-08-17 and 2026-08-18 under mctd_ids 9001-9011. Not a resort, not a camera, not a wait. Present in accuracy_metrics; never publishable. observations produced by a superseded wait conversion — live_crowd rows whose model_regime shows they came from the original naive calc or the M/D/1 formula deleted on 2026-04-04. They measure something the system no longer computes. All fall before the window start; inside the window this rule costs exactly one frame. dates outside 2026-04-05 to 2026-06-10 — Build period before, and off-season replays after. Itemised in scoring_window.excluded_by_the_window.

Generated

read-only recompute from predictions.hourly_predictions / predictions.live_corrected_hourly vs live_crowd + crowd_reports; ground-truth rule and pooling reproduced from engine/Backend/accuracy_metrics.py Generated 2026-08-27.

• MethodologyHow these figures are produced, and what we do not filter.
• Trust CenterEverything else we publish about ourselves.

Figures measured August 27, 2026. Data from the synthetic "simulator" resort is excluded from every number.