# Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

arXiv: https://arxiv.org/abs/2412.05244 · HTML: https://mlsanity.com/paper/2412.05244/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2310.08278/) (2023)
- [Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models](https://mlsanity.com/paper/1909.09020/) (2019)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Wavelet Score-Based Generative Modeling](https://mlsanity.com/paper/2208.05003/) (2022)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)
- [Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation](https://mlsanity.com/paper/2305.00909/) (2023)
- [xVal: A Continuous Numerical Tokenization for Scientific Language Models](https://mlsanity.com/paper/2310.02989/) (2023)
- [First De-Trend then Attend: Rethinking Attention for Time-Series Forecasting](https://mlsanity.com/paper/2212.08151/) (2022)
- [ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning](https://mlsanity.com/paper/2306.06340/) (2023)
- [Wavelet-Based Image Tokenizer for Vision Transformers](https://mlsanity.com/paper/2405.18616/) (2024)
- [Wavelets Are All You Need for Autoregressive Image Generation](https://mlsanity.com/paper/2406.19997/) (2024)

## Built on by (most cited first, 27 of 27)

- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain](https://mlsanity.com/paper/2505.08550/) (2025)
- [Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility](https://mlsanity.com/paper/2510.03358/) (2025)
- [Byte Pair Encoding for Efficient Time Series Forecasting](https://mlsanity.com/paper/2505.14411/) (2025)
- [Wavelet-Driven Masked Multiscale Reconstruction for PPG Foundation Models](https://mlsanity.com/paper/2601.12215/) (2026)
- [Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoning](https://mlsanity.com/paper/2602.07830/) (2026)
- [Rethinking the Role of LLMs in Time Series Forecasting](https://mlsanity.com/paper/2602.14744/) (2026)
- [Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization](https://mlsanity.com/paper/2504.10900/) (2025)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Zeus: Towards Tuning-Free Foundation Model for Time Series Analysis](https://mlsanity.com/paper/2607.01918/) (2026)
- [A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models](https://mlsanity.com/paper/2602.16626/) (2026)
- [WaveMoE: A Wavelet-Enhanced Mixture-of-Experts Foundation Model for Time Series Forecasting](https://mlsanity.com/paper/2604.10544/) (2026)
- [What If We Let Forecasting Forget? A Sparse Bottleneck for Cross-Variable Dependencies](https://mlsanity.com/paper/2605.08289/) (2026)
- [Data-Driven Forecasting of three-Component Seismograms Using Transformer Architectures](https://mlsanity.com/paper/2606.02912/) (2026)
- [BEAT: Balanced Frequency Adaptive Tuning for Long-Term Time-Series Forecasting](https://mlsanity.com/paper/2501.19065/) (2025)
- [Multi-Resolution Analysis of the Convective Structure of Tropical Cyclones for Short-Term Intensity Guidance](https://mlsanity.com/paper/2510.19854/) (2025)
- [There is No "apple" in Timeseries: Rethinking TSFM through the Lens of Invariance](https://mlsanity.com/paper/2510.20119/) (2025)
- [Deep Time-Series Forecasting in 10 Years: A Survey](https://mlsanity.com/paper/2603.19899/) (2026)
- [MsFormer: Enabling Robust Predictive Maintenance Services for Industrial Devices](https://mlsanity.com/paper/2603.23076/) (2026)
- [An Empirical Markov Chain Car-Following (MC-CF) Model](https://mlsanity.com/paper/2603.27909/) (2026)
- [TimeTok: Granularity-Controllable Time-Series Generation via Hierarchical Tokenization](https://mlsanity.com/paper/2605.01418/) (2026)
- [Spectral Vision Transformer for Efficient Tokenization with Limited Data](https://mlsanity.com/paper/2605.12026/) (2026)
- [Dywave: Event-Aligned Dynamic Tokenization for Heterogeneous IoT Sensing Signals](https://mlsanity.com/paper/2605.14014/) (2026)
- [JEPA for AI-Native 6G: Predictive Representations and Open Challenges](https://mlsanity.com/paper/2607.09798/) (2026)
- [Grounding Time-Series Foundation Models in Digital Twin Topology for Predictive Maintenance](https://mlsanity.com/paper/2609.40071/) (2026)
- [W-JEPA: Wavelet Joint-Embedding Predictive Architecture for Wearable Activity Recognition](https://mlsanity.com/paper/2610.19411/) (2026)

## Abstract (arXiv)

How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) provides better accuracy than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and performs on par or better than modern deep learning models trained specifically on each dataset; and ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics. In addition, we show that our method can easily capture complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.