mlsanityData to 7 Oct 2026

Multi-modal Time Series Analysis: A Tutorial and Survey

arXiv 2503.13709 ↗First posted 17 Mar 2025cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Multi-modal time series analysis has recently emerged as a prominent research area in data mining, driven by the increasing availability of diverse data modalities, such as text, images, and structured tabular data from real-world sources. However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise. Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions based on deep learning methods, significantly enhancing various downstream tasks. In this tutorial and survey, we present a systematic and up-to-date overview of multi-modal time series datasets and methods. We first state the existing challenges of multi-modal time series analysis and our motivations, with a brief introduction of preliminaries. Then, we summarize the general pipeline and categorize existing methods through a unified cross-modal interaction framework encompassing fusion, alignment, and transference at different levels (textiti.e., input, intermediate, output), where key concepts and ideas are highlighted. We also discuss the real-world applications of multi-modal analysis for both standard and spatial time series, tailored to general and specific domains. Finally, we discuss future research directions to help practitioners explore and exploit multi-modal time series. The up-to-date resources are provided in the GitHub repository: https://github.com/UConn-DSIS/Multi-modal-Time-Series-Analysis

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 51 of its 97 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

51 of 97 references.

References · 51

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2020Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networkscited by 563
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  7. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  9. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  10. 2017A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Predictioncited by 245
  11. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  12. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
38 more
  1. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  2. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  3. 2024TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignmentcited by 97
  4. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 93
  5. 2023Time Series as Images: Vision Transformer for Irregularly Sampled Time Seriescited by 79
  6. 2024Context is Key: A Benchmark for Forecasting with Essential Textual Informationcited by 78
  7. 2022Spatio-Temporal Self-Supervised Learning for Traffic Flow Predictioncited by 77
  8. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 74
  9. 2024Causal machine learning for predicting treatment outcomescited by 73
  10. 2023Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Datacited by 68
  11. 2021Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classificationcited by 67
  12. 2019ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotationcited by 65
  13. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  14. 2025Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecastingcited by 59
  15. 2021Traffic Flow Forecasting with Spatial-Temporal Graph Diffusion Networkcited by 52
  16. 2024AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Frameworkcited by 47
  17. 2023ChatGPT Informed Graph Neural Network for Stock Movement Predictioncited by 41
  18. 2024Can LLMs Understand Time Series Anomalies?cited by 41
  19. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  20. 2019Using Clinical Notes with Time Series Data for ICU Managementcited by 38
  21. 2025TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecastercited by 36
  22. 2025TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agentscited by 33
  23. 2024How Can Large Language Models Understand Spatial-Temporal Data?cited by 31
  24. 2023Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognitioncited by 27
  25. 2021How to Leverage Multimodal EHR Data for Better Medical Predictions?cited by 26
  26. 2025Harnessing Vision Models for Time Series Analysis: A Surveycited by 26
  27. 2023Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infectioncited by 22
  28. 2024Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotationcited by 21
  29. 2024Towards Explainable Traffic Flow Prediction with Large Language Modelscited by 20
  30. 2024See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzerscited by 20
  31. 2024Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Datacited by 20
  32. 2021Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trendscited by 15
  33. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 15
  34. 2019Adversarial Representation Learning for Robust Patient-Independent Epileptic Seizure Detectioncited by 13
  35. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  36. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Predictioncited by 10
  37. 2024Plots Unlock Time-Series Understanding in Multimodal Modelscited by 9
  38. 2023Multi-Modal Financial Time-Series Retrieval Through Latent Space Projectionscited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 11.

24 papers.

Sort:

By area:

  1. 2025Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Seriescited by 16
  2. 2025From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretizationcited by 13
  3. 2025Diffusion Models for Time Series Forecasting: A Surveycited by 11
  4. 2025VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbonescited by 11
  5. 2025From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?cited by 7
  6. 2026Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecastingcited by 5
  7. 2025Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approachcited by 4
  8. 2025MLLM4TS: Leveraging Vision and Multimodal Language Models for General Time-Series Analysiscited by 4
  9. 2026Bridging the Last Mile of Time Series Forecasting with LLM Agentscited by 3
  10. 2026Post-Training in Time Series Foundation Models: A Unifying Frameworkcited by 3
  11. 2025Towards Interpretable and Trustworthy Time Series Reasoning: A BlueSky Visioncited by 2
  12. 2026Harnessing Generalist Agents for Contextualized Time Seriescited by 2
12 more
  1. 2025Multimodal Remote Inferencecited by 1
  2. 2026Empowering Power Outage Prediction with Spatially Aware Hybrid Graph Neural Networks and Contrastive Learningcited by 1
  3. 2026Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecastingcited by 1
  4. 2025From Time Series Analysis to Question Answering: A Survey in the LLM Eracited by 0
  5. 2025Forecasting Commodity Price Shocks Using Temporal and Semantic Fusion of Prices Signals and Agentic Generative AI Extracted Economic Newscited by 0
  6. 2025MedM2T: A MultiModal Framework for Time-Aware Modeling with Electronic Health Record and Electrocardiogram Datacited by 0
  7. 2025Early GVHD Prediction in Liver Transplantation via Multi-Modal Deep Learning on Imbalanced EHR Datacited by 0
  8. 2026Aura: Universal Multi-dimensional Exogenous Integration for Aviation Time Seriescited by 0
  9. 2026Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoningcited by 0
  10. 2026MetaCaster: Meta-Harness-Optimized Agent for End-to-End Few-Shot Learning of Lightweight Time Series Forecasterscited by 0
  11. 2026Multi-Modal Anomaly Detection: A Surveycited by 0
  12. 2026When Does Context Routing Help? A Systematic Study of Multi-Modal Fusion in Time Series Forecastingcited by 0