Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability
Not on the map. Why?
Abstract (arXiv)
In recent years, time-series question answering (QA) systems have made significant progress. However, generating a correct answer does not show whether retaining the supplied numerical series improves task performance, nor whether the prediction is sensitive to changes in that input. While some systems provide rationales, answer accuracy also does not show whether their numerical claims are grounded in the supplied series or whether the stated inference is valid. In this work, we focus on evaluating four time-series QA systems: TimeOmni-1, ChatTS, TimeOmni-VL, and Time-MQA. First, for three systems with released evaluation data, we reproduce their reported results and compare the performance of the systems with their backbones. Then, we introduce a benchmark named COMMON-TSQA, which collects public evaluation datasets from existing time-series benchmarks and unifies their sample representation, task definitions, and answer schemas, while evaluating each system through its own interface under common evaluation criteria. The evaluation uses the original condition and six interventions while keeping the question and target fixed. Our analysis shows that aggregate performance alone can obscure how systems use numerical evidence. Similar task-level scores can arise despite substantial changes in individual predictions. Some interventions induce simple fallback behavior rather than preserved task ability. We also evaluate rationales for factual grounding, inference validity, and consistency with the final answer. We find that rationales often contain time-series claims unsupported by the input. Moreover, the rationale audit shows that agreement between a rationale and its final answer can coexist with incorrect numerical descriptions or invalid intermediate inferences.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 30 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
27 of 30 references.
References · 27
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
- 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
- 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
- 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
- 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
- 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 100
- 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 88
- 2024Context is Key: A Benchmark for Forecasting with Essential Textual Informationcited by 87
- 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 82
15 more
- 2026It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarkscited by 33
- 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
- 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 26
- 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
- 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 15
- 2025When Does Multimodality Lead to Better Time Series Forecasting?cited by 11
- 2026TSAQA: Time Series Analysis Question And Answering Benchmarkcited by 11
- 2026RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Modelscited by 11
- 2025CaTS-Bench: Can Language Models Describe Time Series?cited by 10
- 2026TimeOmni-VL: Unified Models for Time Series Understanding and Generationcited by 7
- 2026Adaptive Time Series Reasoning via Segment Selectioncited by 4
- 2026PATRA: Pattern-Aware Alignment and Balanced Reasoning for Time Series Question Answeringcited by 4
- 2026TSQueryBench: LLM-as-a-Judge for Time Series Explanationscited by 4
- 2026SenTSR-Bench: Thinking with Injected Knowledge for Time-Series Reasoningcited by 2
- 2026FactoryBench: Evaluating Industrial Machine Understandingcited by 2
Built on by
Not cited yet.