mlsanityData to 14 Oct 2026

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

arXiv 2610.17794 ↗First posted 13 Oct 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external conditions, and expert expectations. However, reliable evaluation remains difficult: existing benchmarks often have limited domain coverage, lack standardized history-future splits, or provide text that is not guaranteed to inform the future. We introduce GIFT-Eval-CTX, a benchmark that isolates whether forecasting methods can translate verified future information into accurate numerical trajectories. Built upon GIFT-Eval, GIFT-Eval-CTX contains 10,000 instances across 20 datasets. Each instance provides structured effects derived from the realized future: Macro effects describe horizon-wide behavior, while Micro effects specify localized changes. We construct these annotations with an agentic pipeline combining LLM-based discovery and deterministic code validation. We benchmark TSFMs, frontier LLMs, and tool-using agentic systems. The results show that structured effects reduce aggregate relative MASE by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8 relative to history-only prompting. The evaluated MoiraiAgent system achieves the best overall result. We further analyze the contributions of structured effects and agentic systems to performance, and identify opportunities for future research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 34 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

31 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks5 Feb

Other references · 30

  1. 2024Chronos: Learning the Language of Time Seriescited by 759
  2. 2023A decoder-only foundation model for time-series forecastingcited by 692
  3. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  4. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  5. 2025Chronos-2: From Univariate to Universal Forecastingcited by 194
  6. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  7. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  8. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
  9. 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 100
  10. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  11. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 88
  12. 2024Context is Key: A Benchmark for Forecasting with Essential Textual Informationcited by 87
18 more
  1. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 82
  2. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 74
  3. 2025Moirai 2.0: When Less Is More for Time Series Forecastingcited by 69
  4. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 55
  5. 2025OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Datacited by 36
  6. 2026It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarkscited by 33
  7. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 30
  8. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  9. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
  10. 2025BEDTime: A Unified Benchmark for Automatically Describing Time Seriescited by 12
  11. 2025FinMultiTime: A Four-Modal Bilingual Dataset for Financial Time-Series Analysiscited by 11
  12. 2025Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecastingcited by 10
  13. 2026MMTS-BENCH: A Comprehensive Benchmark for Time Series Understanding and Reasoningcited by 9
  14. 2025MoTime: A Dataset Suite for Multimodal Time Series Forecastingcited by 7
  15. 2026TimeOmni-VL: Unified Models for Time Series Understanding and Generationcited by 7
  16. 2026FinTexTS: Financial Text-Paired Time-Series Dataset via Semantic-Based and Multi-Level Pairingcited by 5
  17. 2026Thoth: Mid-Training Bridges LLMs to Time Series Understandingcited by 4
  18. 2026Empowering Time Series Analysis with Large-Scale Multimodal Pretrainingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecasting
  2. 2026CastFSR: A Fast--Slow--Reflect Agentic Reasoning Framework for Context-Aware Time Series Forecasting
  3. 2026TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting
  4. 2025TimeCopilot
  5. 2026TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems
  6. 2026Position: Beyond Model-Centric Prediction -- Agentic Time Series Forecasting
  7. 2026When Does Context Routing Help? A Systematic Study of Multi-Modal Fusion in Time Series Forecasting
  8. 2026OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning