# From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting

arXiv: https://arxiv.org/abs/2403.11047 · HTML: https://mlsanity.com/paper/2403.11047/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 17 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Forecasting with time series imaging](https://mlsanity.com/paper/1904.08064/) (2019)
- [Financial Time Series Forecasting using CNN and Transformer](https://mlsanity.com/paper/2304.04912/) (2023)
- [Visual Time Series Forecasting: An Image-driven Approach](https://mlsanity.com/paper/2011.09052/) (2020)
- [Trading via Image Classification](https://mlsanity.com/paper/1907.10046/) (2019)
- [Deep Autoregressive Models with Spectral Attention](https://mlsanity.com/paper/2107.05984/) (2021)
- [Nonstationary Portfolios: Diversification in the Spectral Domain](https://mlsanity.com/paper/2102.00477/) (2021)
- [Deep Video Prediction for Time Series Forecasting](https://mlsanity.com/paper/2102.12061/) (2021)
- [Dynamic Portfolio Cuts: A Spectral Approach to Graph-Theoretic Diversification](https://mlsanity.com/paper/2106.03417/) (2021)

## Built on by (most cited first, 9 of 9)

- [Harnessing Vision Models for Time Series Analysis: A Survey](https://mlsanity.com/paper/2502.08869/) (2025)
- [Harnessing Vision-Language Models for Time Series Anomaly Detection](https://mlsanity.com/paper/2506.06836/) (2025)
- [Multi-Modal View Enhanced Large Vision Models for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2505.24003/) (2025)
- [From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?](https://mlsanity.com/paper/2505.24030/) (2025)
- [TADACap: Time-series Adaptive Domain-Aware Captioning](https://mlsanity.com/paper/2504.11441/) (2025)
- [Unsupervised Time-Series Signal Analysis with Autoencoders and Vision Transformers: A Review of Architectures and Applications](https://mlsanity.com/paper/2504.16972/) (2025)
- [SSDA: Bridging Spectral and Structural Gaps via Dual Adaptation for Vision-Based Time Series Forecasting](https://mlsanity.com/paper/2605.12550/) (2026)
- [Vision4PPG: Emergent PPG Analysis Capability of Vision Foundation Models for Vital Signs like Blood Pressure](https://mlsanity.com/paper/2510.10366/) (2025)
- [Visual Chart Representations for Cryptocurrency Regime Prediction: A Systematic Deep Learning Study](https://mlsanity.com/paper/2605.00875/) (2026)

## Abstract (arXiv)

Time series forecasting plays a crucial role in decision-making across various domains, but it presents significant challenges. Recent studies have explored image-driven approaches using computer vision models to address these challenges, often employing lineplots as the visual representation of time series data. In this paper, we propose a novel approach that uses time-frequency spectrograms as the visual representation of time series data. We introduce the use of a vision transformer for multimodal learning, showcasing the advantages of our approach across diverse datasets from different domains. To evaluate its effectiveness, we compare our method against statistical baselines (EMA and ARIMA), a state-of-the-art deep learning-based approach (DeepAR), other visual representations of time series data (lineplot images), and an ablation study on using only the time series as input. Our experiments demonstrate the benefits of utilizing spectrograms as a visual representation for time series data, along with the advantages of employing a vision transformer for simultaneous learning in both the time and frequency domains.