# Distilling Time Series Foundation Models for Efficient Forecasting

arXiv: https://arxiv.org/abs/2601.12785 · HTML: https://mlsanity.com/paper/2601.12785/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [DUET: Dual Clustering Enhanced Multivariate Time Series Forecasting](https://mlsanity.com/paper/2412.10859/) (2024)
- [TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment](https://mlsanity.com/paper/2406.01638/) (2024)
- [CLIP-KD: An Empirical Study of CLIP Model Distillation](https://mlsanity.com/paper/2307.12732/) (2023)
- [Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting](https://mlsanity.com/paper/2507.02939/) (2025)
- [Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2207.05273/) (2022)
- [Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillation](https://mlsanity.com/paper/2505.02138/) (2025)
- [An Accurate and Interpretable Framework for Trustworthy Process Monitoring](https://mlsanity.com/paper/2302.10426/) (2023)
- [BLAST: Balanced Sampling Time Series Corpus for Universal Forecasting Models](https://mlsanity.com/paper/2505.17871/) (2025)
- [TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation](https://mlsanity.com/paper/2502.15016/) (2025)
- [Training-free LLM Merging for Multi-task Learning](https://mlsanity.com/paper/2506.12379/) (2025)
- [Wavelet-Inspired Multiscale Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2401.06040/) (2024)
- [EnergyPatchTST: Multi-scale Time Series Transformers with Uncertainty Estimation for Energy Forecasting](https://mlsanity.com/paper/2508.05454/) (2025)
- [TimeMKG: Knowledge-Infused Causal Reasoning for Multivariate Time Series Modeling](https://mlsanity.com/paper/2508.09630/) (2025)

## Built on by (most cited first, 11 of 11)

- [Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation](https://mlsanity.com/paper/2605.09253/) (2026)
- [Post-Training in Time Series Foundation Models: A Unifying Framework](https://mlsanity.com/paper/2607.20002/) (2026)
- [GaitProtector: Impersonation-Driven Gait De-Identification via Training-Free Diffusion Latent Optimization](https://mlsanity.com/paper/2605.12431/) (2026)
- [History-informed Lagrangian Neural Networks](https://mlsanity.com/paper/2608.13215/) (2026)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations](https://mlsanity.com/paper/2604.14246/) (2026)
- [A Checklist to assess the energy and carbon impacts of ML/AI applications in Earth System Modeling](https://mlsanity.com/paper/2609.00847/) (2026)
- [Decision-Oriented Uncertainty Quantification for Risk Control in Earth System Spatiotemporal Foundation Models](https://mlsanity.com/paper/2609.14821/) (2026)
- [MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making](https://mlsanity.com/paper/2609.14823/) (2026)
- [AsyncCouple-Flow: Asynchronous Cross-Modal Coupling and Flow Matching for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2609.16573/) (2026)
- [CoRe: Coherence and Relational Alignment for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2609.19670/) (2026)

## Abstract (arXiv)

Time Series foundation models (TSFMs) deliver strong forecasting performance through large-scale pretraining, but their large parameter sizes make deployment costly. While knowledge distillation offers a natural and effective approach for model compression, techniques developed for general machine learning tasks are not directly applicable to time series forecasting due to the unique characteristics. To address this, we present DistilTS, the first distillation framework specifically designed for TSFMs. DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting. To overcome these issues, DistilTS introduces horizon-weighted objectives to balance learning across horizons, and a temporal alignment strategy that reduces architectural mismatch, enabling compact models. Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x. Code is available at: https://github.com/itsnotacie/DistilTS-ICASSP2026.