mlsanityData to 14 Oct 2026

From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol

arXiv 2606.24996 ↗First posted 23 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Forecasting leaderboards rank models by predictive quality, but their winners are often read as deployment-ready top-1 advice. That reading can fail when forecasts are passed through a fixed decision interface, such as an alert threshold, a top-k budget, or a switching-cost policy. We study when a forecast-side winner can be certified as deployment-actionable for a specified interface and deployed utility. We introduce a fail-closed certification protocol whose gates are sufficient evidential conditions for a strong claim: a friction-caused, non-tie, statistically supported, and recurrent deployment-side reversal. Traffic-Hourly provides a certified anchor: winners agree at zero friction, but positive switching friction makes the forecast winner deployed-suboptimal. A locked native audit tests overclaiming: across 22 verified candidates and 362 full-grid cells, 155 apparent forecast/deployment winner inversions are blocked before certification. The contribution is not a new forecaster, metric, or universal utility, but a conservative protocol for deciding when forecasting leaderboard winners should be read as deployment-actionable top-1 advice.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

17 of 23 references.

References · 17

  1. 2023A decoder-only foundation model for time-series forecastingcited by 692
  2. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  3. 2017Smart "Predict, then Optimize"cited by 339
  4. 2021Monash Time Series Forecasting Archivecited by 267
  5. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  6. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  7. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 130
  8. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  9. 2024Context is Key: A Benchmark for Forecasting with Essential Textual Informationcited by 87
  10. 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 64
  11. 2024TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecastingcited by 58
  12. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 40
5 more
  1. 2025Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarkscited by 10
  2. 2025TimeRecipe: A Time-Series Forecasting Recipe via Benchmarking Module Level Effectivenesscited by 7
  3. 2026Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluationcited by 6
  4. 2026TempusBench: An Evaluation Framework for Time-Series Forecastingcited by 4
  5. 2025Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Taskscited by 3

Built on by

Not cited yet.