# Towards Interpretable Time Series Foundation Models

arXiv: https://arxiv.org/abs/2507.07439 · HTML: https://mlsanity.com/paper/2507.07439/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Time series reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Placed in

- LLM reasoning / [Time series reasoning](https://mlsanity.com/lr/time-series-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [UniTS: A Unified Multi-Task Time Series Model](https://mlsanity.com/paper/2403.00131/) (2024)
- [Position: What Can Large Language Models Tell Us about Time Series Analysis](https://mlsanity.com/paper/2402.02713/) (2024)
- [Can LLMs Understand Time Series Anomalies?](https://mlsanity.com/paper/2410.05440/) (2024)
- [Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark](https://mlsanity.com/paper/2404.16563/) (2024)
- [The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models](https://mlsanity.com/paper/2309.06236/) (2023)
- [Explaining Agent-Based Financial Market Simulation](https://mlsanity.com/paper/1909.11650/) (2019)
- [Multi-Modal Financial Time-Series Retrieval Through Latent Space Projections](https://mlsanity.com/paper/2309.16741/) (2023)

## Built on by (most cited first, 3 of 3)

- [Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2604.28149/) (2026)
- [Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models](https://mlsanity.com/paper/2603.10071/) (2026)
- [Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation](https://mlsanity.com/paper/2609.13199/) (2026)

## Abstract (arXiv)

In this paper, we investigate the distillation of time series reasoning capabilities into small, instruction-tuned language models as a step toward building interpretable time series foundation models. Leveraging a synthetic dataset of mean-reverting time series with systematically varied trends and noise levels, we generate natural language annotations using a large multimodal model and use these to supervise the fine-tuning of compact Qwen models. We introduce evaluation metrics that assess the quality of the distilled reasoning - focusing on trend direction, noise intensity, and extremum localization - and show that the post-trained models acquire meaningful interpretive capabilities. Our results highlight the feasibility of compressing time series understanding into lightweight, language-capable models suitable for on-device or privacy-sensitive deployment. This work contributes a concrete foundation toward developing small, interpretable models that explain temporal patterns in natural language.