mlsanityData to 14 Oct 2026

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset

arXiv 2506.20093 ↗First posted 25 Jun 2025cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Time-series data are critical in diverse applications, such as industrial monitoring, medical diagnostics, and climate research. However, effectively integrating these high-dimensional temporal signals with natural language for dynamic, interactive tasks remains a significant challenge. To address this, we introduce the Time-Series Question Answering (Time-Series QA) task and release EngineMT-QA, the first large-scale, multi-task, temporal-textual QA dataset designed to capture complex interactions between time-series signals and natural language. Building on this resource, we propose the Instruct Time Transformer (ITFormer), a novel framework that bridges time-series encoders with frozen large language models (LLMs). ITFormer effectively extracts, aligns, and fuses temporal and textual features, achieving a strong improvement in QA accuracy over strong baselines with fewer than 1% additional trainable parameters. By combining computational efficiency with robust cross-modal modeling, our work establishes a adaptable paradigm for integrating temporal data with natural language, paving the way for new research and applications in multi-modal AI. More details about the project, including datasets and code, are available at: https://pandalin98.github.io/itformersite/

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

28 of 53 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  12. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
16 more
  1. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  3. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  4. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  5. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2016Visual Dialogcited by 377
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  9. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  10. 2023A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detectioncited by 145
  11. 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 100
  12. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 98
  13. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 94
  14. 2024TimeSeriesExam: A time series understanding examcited by 49
  15. 2024Position: What Can Large Language Models Tell Us about Time Series Analysiscited by 47
  16. 2025Leveraging Large Self-Supervised Time-Series Models for Transferable Diagnosis in Cross-Aircraft Type Bleed Air Systemcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 29.

37 papers.

Sort:

By area:

  1. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  2. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  3. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
  4. 2025BEDTime: A Unified Benchmark for Automatically Describing Time Seriescited by 12
  5. 2026TSAQA: Time Series Analysis Question And Answering Benchmarkcited by 11
  6. 2026STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learningcited by 7
  7. 2026TimeOmni-VL: Unified Models for Time Series Understanding and Generationcited by 7
  8. 2026TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scalecited by 6
  9. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 6
  10. 2026Thoth: Mid-Training Bridges LLMs to Time Series Understandingcited by 4
  11. 2026Learning Transferable Sensor Models via Language-Informed Pretrainingcited by 4
  12. 2026PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Toolscited by 4
25 more
  1. 2026CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Seriescited by 4
  2. 2026TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoningcited by 2
  3. 2026FactoryBench: Evaluating Industrial Machine Understandingcited by 2
  4. 2025Chain-of-thought Reviewing and Correction for Time Series Question Answeringcited by 1
  5. 2026Multi-Modal Time Series Prediction via Mixture of Modulated Expertscited by 1
  6. 2026CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Trafficcited by 1
  7. 2026DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rulescited by 1
  8. 2026Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoningcited by 1
  9. 2026Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Languagecited by 1
  10. 2025DiffNator: Generating Structured Explanations of Time-Series Differencescited by 0
  11. 2025Can VLM Pseudo-Labels Train a Time-Series QA Model That Outperforms the VLM?cited by 0
  12. 2025Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Datacited by 0
  13. 2025TimeSeries2Report prompting enables adaptive large language model management of lithium-ion batteriescited by 0
  14. 2026ChatAD: Reasoning-Enhanced Time-Series Anomaly Detection with Multi-Turn Instruction Evolutioncited by 0
  15. 2026Sonar-TS: Search-Then-Verify Natural Language Querying for Time Series Databasescited by 0
  16. 2026KairosVL: Orchestrating Time Series and Semantics for Unified Reasoningcited by 0
  17. 2026IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenancecited by 0
  18. 2026Beyond Uniform Tokens: Adaptive Compression for Time Series Language Modelscited by 0
  19. 2026Beyond Tokenization: Direct Timestep Embedding and Contrastive Alignment for Time-Series Question Answeringcited by 0
  20. 2026PhysioBench: A Unified Benchmark for Physiological Signal Question Answeringcited by 0
  21. 2026ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMscited by 0
  22. 2026OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoningcited by 0
  23. 2026FreSia: Frequency-Semantic Instantiation and Alignment for Multivariate Time Series Analysiscited by 0
  24. 2026Lapras: Latent Reasoning for Time Series Language Modelscited by 0
  25. 2026MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Visioncited by 0