# TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation

arXiv: https://arxiv.org/abs/2410.05711 · HTML: https://mlsanity.com/paper/2410.05711/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2304.05206/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Deconstructing Denoising Diffusion Models for Self-Supervised Learning](https://mlsanity.com/paper/2401.14404/) (2024)
- [Non-autoregressive Conditional Diffusion Models for Time Series Prediction](https://mlsanity.com/paper/2306.05043/) (2023)
- [Diffusion Models for Time Series Applications: A Survey](https://mlsanity.com/paper/2305.00624/) (2023)
- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)
- [The Rise of Diffusion Models in Time-Series Forecasting](https://mlsanity.com/paper/2401.03006/) (2024)
- [SDWPF: A Dataset for Spatial Dynamic Wind Power Forecasting Challenge at KDD Cup 2022](https://mlsanity.com/paper/2208.04360/) (2022)
- [FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification](https://mlsanity.com/paper/2302.09818/) (2023)
- [TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling](https://mlsanity.com/paper/2402.02475/) (2024)

## Built on by (most cited first, 14 of 14)

- [From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretization](https://mlsanity.com/paper/2508.09191/) (2025)
- [MemCast: Memory-Driven Time Series Forecasting with Experience-Conditioned Reasoning](https://mlsanity.com/paper/2602.03164/) (2026)
- [DisenTS: Disentangled Channel Evolving Pattern Modeling for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.22981/) (2024)
- [Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach](https://mlsanity.com/paper/2604.06727/) (2026)
- [Sequence Diffusion Model for Temporal Link Prediction in Continuous-Time Dynamic Graph](https://mlsanity.com/paper/2601.23233/) (2026)
- [StaTS: Spectral Trajectory Schedule Learning for Adaptive Time Series Forecasting with Frequency Guided Denoiser](https://mlsanity.com/paper/2603.00037/) (2026)
- [TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition](https://mlsanity.com/paper/2608.15861/) (2026)
- [CoGenCast: A Coupled Autoregressive-Flow Generative Framework for Time Series Forecasting](https://mlsanity.com/paper/2602.03564/) (2026)
- [Deep Time-Series Forecasting in 10 Years: A Survey](https://mlsanity.com/paper/2603.19899/) (2026)
- [When Does Self-Supervised Learning Transfer to Time-Series Tasks?](https://mlsanity.com/paper/2605.19462/) (2026)
- [VLBM: Variational Latent Basis Modeling for OOD Robust Multivariate Time Series Forecasting](https://mlsanity.com/paper/2606.02138/) (2026)
- [GLAIM: Learning Global and Local Adaptive Inter-Variable Dependency for Multivariate Time Series Imputation](https://mlsanity.com/paper/2608.02366/) (2026)
- [WinoTS: Wavelet-based Self-Distillation for Time Series Models](https://mlsanity.com/paper/2609.39337/) (2026)
- [ProtoFlow: Prototype-Guided Flow Matching for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2610.01320/) (2026)

## Abstract (arXiv)

Self-supervised learning has garnered increasing attention in time series analysis for benefiting various downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively capture both long-term dynamic evolution and subtle local patterns in a unified manner. In this work, we propose TimeDART, a novel self-supervised time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. Specifically, we first employ a causal Transformer encoder, accompanied by a patch-based embedding strategy, to model the evolving trends from left to right. Building on this global modeling, we further introduce a denoising diffusion process to capture fine-grained local patterns through forward diffusion and reverse denoising. Finally, we optimize the model in an autoregressive manner. As a result, TimeDART effectively accounts for both global and local sequence features in a coherent way. We conduct extensive experiments on public datasets for time series forecasting and classification. The experimental results demonstrate that TimeDART consistently outperforms previous compared methods, validating the effectiveness of our approach. Our code is available at https://github.com/Melmaphother/TimeDART.