A Covariate You Couldn't Have Known: Foresight Leakage in Forecasting Benchmarks
July 29, 2026
The specialist won 26 of 29 cities. Then I stopped telling it tomorrow’s weather.
This post is about one line in a data loader, and how much it changed a result.
I was comparing a zero-shot time-series foundation model (Chronos-Bolt) against a tuned LightGBM specialist on hourly forecasting in 29 cities, in two domains: PM2.5 air pollution and 2 m temperature. The forecast horizon was 24 hours and the metric was MASE (mean absolute scaled error, where lower is better and 1.0 means “no better than a seasonal-naive baseline”).
On PM2.5 the two were close. On temperature the specialist looked dominant:
| Temperature, 24 h ahead | Panel-mean MASE |
|---|---|
| LightGBM specialist | 0.533 |
| Chronos-Bolt, zero-shot | 0.792 |
The specialist won 26 of 29 cities (Wilcoxon P = 1.4 × 10⁻⁶). There were six city-level wins for the specialist that survived false-discovery control, and none the other way. It was tempting to write a tidy “foundation models are fine for pollution, specialists still win for weather” conclusion.
Where the advantage came from#
The specialist uses weather covariates: humidity, pressure, solar radiation, dew point. Most forecasting libraries let you pass these as future_covariates, and the usual benchmark convention fills them with the observed values at the forecast target time.
Think about what that means at 24 hours ahead. To forecast temperature at 3 pm tomorrow, the model is handed tomorrow’s 3 pm humidity, pressure and radiation. Those are close to deterministic physical drivers of temperature. The model is being given a perfect weather forecast of its own inputs, which no operator has.
For PM2.5 this barely matters, because weather is a weak 24-hour predictor of pollution. For temperature it matters a great deal.
The ablation#
I re-ran the specialist with causal covariates: each weather variable enters at its last value known at the forecast origin. Calendar features stay future-known, since a calendar is deterministic. I logged this ablation in the study’s deviations record before running it, and I re-ran the full 29-city panel in both domains, not a convenient subset.
| Temperature, 24 h ahead | MASE | Cities won vs Chronos-Bolt |
|---|---|---|
| Specialist, perfect-foresight covariates | 0.533 | 26/29 (P = 1.4 × 10⁻⁶) |
| Specialist, causal covariates | 0.745 | 16/29 (P = 0.29) |
| Chronos-Bolt, zero-shot (no covariates) | 0.792 |
The foresight assumption alone was worth +0.212 MASE (P = 2.6 × 10⁻⁸). The six FDR-significant specialist wins became zero, and the foundation model picked up one.
PM2.5 hardly moved: 0.662 with foresight, 0.692 causal, both statistically indistinguishable from Chronos-Bolt at 0.662.
Two things I had to be careful about#
Non-significant is not the same as equivalent. After the ablation the specialist still had a small edge on temperature (0.745 vs 0.792) that was not significant. It would be easy to write “the models perform the same.” A non-significant test cannot show that. So I ran paired equivalence tests (TOST) at a margin of 0.05 MASE. On temperature under causal covariates equivalence was not established, and the 90% interval of the difference, -0.114 to 0.018, leans in the specialist’s favour. The honest summary is “no significant difference, and the remaining uncertainty tilts toward the specialist”, not “a tie”.
Causal covariates are a floor, not reality. Real operators do not hold the weather flat. They take covariates from a numerical weather forecast, which is imperfect but much better than last-known values. The two regimes bracket deployment. Neither describes it.
Putting a real forecast in between#
To locate the deployable point I measured the actual error of archived 24-hour weather forecasts for every covariate the specialist uses, then injected error of that measured size into the true covariates, as an AR(1) process matched to the measured magnitude, bias and lag-1 autocorrelation. A scale factor α sweeps from 0 (perfect foresight) upward, and α = 1 corresponds to real forecast quality.
| Covariate quality | Specialist MASE | Lead over zero-shot | Share of foresight lead kept |
|---|---|---|---|
| Perfect foresight (α = 0) | 0.533 | +0.260 | 100% |
| Measured forecast error (α = 1) | 0.638 | +0.154 | 59% |
| Last known value | 0.745 | +0.048 | 18% |
So the benchmark convention did not invent the specialist’s temperature advantage. It inflated it. A deployment with a trustworthy weather feed should expect to keep roughly three-fifths of the benchmark lead; one without a feed should expect to lose most of it.
Two sanity checks made me trust this more. Repeating the noise injection gave panel means that varied by only 0.009 to 0.018 MASE, small next to the 0.106 shift from perfect to realistic covariates. And the tiers that use no covariates came out bitwise identical at every α, so the degradation only touched the path it was meant to.
One subtlety: matching the size of last-known error does not reproduce its effect. The panel did not reach the last-known MASE of 0.745 until α ≈ 3.4 by extrapolation, far beyond the 1.54 that matching error variance would predict. Holding a covariate flat for 24 hours wipes out the daily cycle. Autocorrelated noise of the same variance blurs that cycle but keeps it. So I report the last-known figure directly rather than reading it off the α curve.
A checklist I now run on any forecasting benchmark#
These are the questions I ask of my own pipelines now, and of papers I read.
- At the forecast origin, which inputs would actually be known? Go through them one by one. Calendar, yes. Observed weather at the target time, no.
- Does the library default fill future covariates with observations? Many do, because that is convenient for backtesting. It is a modelling assumption, and it should be written down as one.
- Is the gap between models larger in domains where covariates nearly determine the target? If a specialist’s advantage is concentrated exactly where its inputs are most informative, suspect foresight before skill.
- Is there a realistic middle regime? If archived forecasts exist for the covariates, use their measured error. Otherwise, report both perfect-foresight and last-known results as a bracket.
- Are the “no difference” claims backed by an equivalence test? If not, say “not significantly different” and stop there.
- Is the perfect-foresight number still in the paper? It should be, clearly labelled as an upper bound. Deleting it hides how much the assumption was worth.
Limits#
- The bracket depends on how informative the covariates are. On PM2.5 the gap between foresight and last-known was about 0.03 MASE, so there was almost nothing to correct. The size of this effect is domain-specific.
- I measured forecast error from one archive of 24-hour forecasts. Covariate quality in other regions, or at longer horizons, will differ.
- The forecast archive covered only 15 of the 29 cities, and its lead times (24 to 47 hours) run past my horizon, so error was calibrated on the covered cities and regressed on lead time. The α = 1 point is an estimate, not a direct measurement for every city.
- This is one specialist (LightGBM) and one foundation-model family in the main panel, with two other foundation models as a generality check. A different specialist might use covariates differently.
The broader point applies well beyond this paper. Any benchmark that feeds models observed future covariates is measuring performance under a perfect forecast of those covariates. That can be a useful ceiling. It should not be reported as the result.
The paper, including the equivalence tests and the covariate-quality sweep, is summarised on the publication page, with the preprint at Research Square.