# Can Multimodal LLMs Perform Time Series Anomaly Detection?

arXiv: https://arxiv.org/abs/2502.17812 · HTML: https://mlsanity.com/paper/2502.17812/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 47 of its 87 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 47)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Deep Learning for Time Series Anomaly Detection: A Survey](https://mlsanity.com/paper/2211.05244/) (2022)
- [BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions](https://mlsanity.com/paper/2308.09936/) (2023)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data](https://mlsanity.com/paper/2412.11376/) (2024)
- [VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2408.17253/) (2024)
- [Local Evaluation of Time Series Anomaly Detection Algorithms](https://mlsanity.com/paper/2206.13167/) (2022)
- [Time Series as Images: Vision Transformer for Irregularly Sampled Time Series](https://mlsanity.com/paper/2303.12799/) (2023)
- [LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting](https://mlsanity.com/paper/2402.16132/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Series](https://mlsanity.com/paper/2012.00168/) (2020)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [ChatGPT Informed Graph Neural Network for Stock Movement Prediction](https://mlsanity.com/paper/2306.03763/) (2023)

## Built on by (most cited first, 12 of 12)

- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Harnessing Vision-Language Models for Time Series Anomaly Detection](https://mlsanity.com/paper/2506.06836/) (2025)
- [JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model](https://mlsanity.com/paper/2504.03770/) (2025)
- [Agentic AI for autonomous anomaly management in complex systems](https://mlsanity.com/paper/2507.15676/) (2025)
- [CALM: A Framework for Continuous, Adaptive, and LLM-Mediated Anomaly Detection in Time-Series Streams](https://mlsanity.com/paper/2508.21273/) (2025)
- [Detecting Time Series Anomalies Like an Expert: A Multi-Agent LLM Framework with Specialized Analyzers](https://mlsanity.com/paper/2605.05725/) (2026)
- [Evaluating Simplification Algorithms for Interpretability of Time Series Classification](https://mlsanity.com/paper/2505.08846/) (2025)
- [ViTs: Teaching Machines to See Time Series Anomalies Like Human Experts](https://mlsanity.com/paper/2510.04710/) (2025)
- [From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning](https://mlsanity.com/paper/2601.21436/) (2026)
- [AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detection](https://mlsanity.com/paper/2602.08868/) (2026)
- [VETime: Vision Enhanced Zero-Shot Time Series Anomaly Detection](https://mlsanity.com/paper/2602.16681/) (2026)
- [Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptation](https://mlsanity.com/paper/2610.02353/) (2026)

## Abstract (arXiv)

Time series anomaly detection (TSAD) has been a long-standing pillar problem in Web-scale systems and online infrastructures, such as service reliability monitoring, system fault diagnosis, and performance optimization. Large language models (LLMs) have demonstrated unprecedented capabilities in time series analysis, the potential of multimodal LLMs (MLLMs), particularly vision-language models, in TSAD remains largely under-explored. One natural way for humans to detect time series anomalies is through visualization and textual description. It motivates our research question: Can multimodal LLMs perform time series anomaly detection? Existing studies often oversimplify the problem by treating point-wise anomalies as special cases of range-wise ones or by aggregating point anomalies to approximate range-wise scenarios. They limit our understanding for realistic scenarios such as multi-granular anomalies and irregular time series. To address the gap, we build a VisualTimeAnomaly benchmark to comprehensively investigate zero-shot capabilities of MLLMs for TSAD, progressively from point-, range-, to variate-wise anomalies, and extends to irregular sampling conditions. Our study reveals several key insights in multimodal MLLMs for TSAD. Built on these findings, we propose a MLLMs-based multi-agent framework TSAD-Agents to achieve automatic TSAD. Our framework comprises scanning, planning, detection, and checking agents that synergistically collaborate to reason, plan, and self-reflect to enable automatic TSAD. These agents adaptively invoke tools such as traditional methods and MLLMs and dynamically switch between text and image modalities to optimize detection performance.