# Time Series as Images: Vision Transformer for Irregularly Sampled Time Series

arXiv: https://arxiv.org/abs/2303.12799 · HTML: https://mlsanity.com/paper/2303.12799/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)

## Built on by (most cited first, 30 of 79)

- [Language Models Still Struggle to Zero-shot Reason about Time Series](https://mlsanity.com/paper/2404.11757/) (2024)
- [VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2408.17253/) (2024)
- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series](https://mlsanity.com/paper/2410.19538/) (2024)
- [TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop](https://mlsanity.com/paper/2503.01013/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Harnessing Vision Models for Time Series Analysis: A Survey](https://mlsanity.com/paper/2502.08869/) (2025)
- [Trajectory Flow Matching with Applications to Clinical Time Series Modeling](https://mlsanity.com/paper/2410.21154/) (2024)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2405.02358/) (2024)
- [See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzers](https://mlsanity.com/paper/2411.02465/) (2024)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data](https://mlsanity.com/paper/2508.02879/) (2025)
- [Can Multimodal LLMs Perform Time Series Anomaly Detection?](https://mlsanity.com/paper/2502.17812/) (2025)
- [VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones](https://mlsanity.com/paper/2508.04379/) (2025)
- [Harnessing Vision-Language Models for Time Series Anomaly Detection](https://mlsanity.com/paper/2506.06836/) (2025)
- [TimeCHEAT: A Channel Harmony Strategy for Irregularly Sampled Multivariate Time Series Analysis](https://mlsanity.com/paper/2412.12886/) (2024)
- [Multi-Modal View Enhanced Large Vision Models for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2505.24003/) (2025)
- [EEO-TFV: Escape-Explore Optimizer for Web-Scale Time-Series Forecasting and Vision Analysis](https://mlsanity.com/paper/2602.02551/) (2026)
- [TimEHR: Image-based Time Series Generation for Electronic Health Records](https://mlsanity.com/paper/2402.06318/) (2024)
- [LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models](https://mlsanity.com/paper/2404.01165/) (2024)
- [From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?](https://mlsanity.com/paper/2505.24030/) (2025)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [A Diffusion Model for Regular Time Series Generation from Irregular Data with Completion and Masking](https://mlsanity.com/paper/2510.06699/) (2025)
- [AimTS: Augmented Series and Image Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2504.09993/) (2025)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [Unifying Prediction and Explanation in Time-Series Transformers via Shapley-based Pretraining](https://mlsanity.com/paper/2501.15070/) (2025)
- [MLLM4TS: Leveraging Vision and Multimodal Language Models for General Time-Series Analysis](https://mlsanity.com/paper/2510.07513/) (2025)
- [MUSE-Net: Missingness-aware mUlti-branching Self-attention Encoder for Irregular Longitudinal Electronic Health Records](https://mlsanity.com/paper/2407.00840/) (2024)
- [Irregularity-Informed Time Series Analysis: Adaptive Modelling of Spatial and Temporal Dynamics](https://mlsanity.com/paper/2410.12257/) (2024)

## Abstract (arXiv)

Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pre-trained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of 42.8% in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST