# Exploring Representations and Interventions in Time Series Foundation Models

arXiv: https://arxiv.org/abs/2409.12915 · HTML: https://mlsanity.com/paper/2409.12915/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)
- [Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers](https://mlsanity.com/paper/2109.08406/) (2021)
- [Weak Supervision for Affordable Modeling of Electrocardiogram Data](https://mlsanity.com/paper/2201.02936/) (2022)
- [Implicit Reasoning in Deep Time Series Forecasting](https://mlsanity.com/paper/2409.10840/) (2024)

## Built on by (most cited first, 20 of 20)

- [Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models](https://mlsanity.com/paper/2503.11411/) (2025)
- [ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response](https://mlsanity.com/paper/2604.21199/) (2026)
- [On the Internal Semantics of Time-Series Foundation Models](https://mlsanity.com/paper/2511.15324/) (2025)
- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [time2time: Causal Intervention in Hidden States to Simulate Rare Events in Time Series Foundation Models](https://mlsanity.com/paper/2509.05801/) (2025)
- [Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function](https://mlsanity.com/paper/2606.12917/) (2026)
- [Investigating Compositional Reasoning in Time Series Foundation Models](https://mlsanity.com/paper/2502.06037/) (2025)
- [The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Models](https://mlsanity.com/paper/2511.07237/) (2025)
- [TabPFN Through The Looking Glass: An interpretability study of TabPFN and its internal representations](https://mlsanity.com/paper/2601.08181/) (2026)
- [Time-Prompt: Integrated Heterogeneous Prompts for Unlocking LLMs in Time Series Forecasting](https://mlsanity.com/paper/2506.17631/) (2025)
- [Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detection](https://mlsanity.com/paper/2509.12650/) (2025)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits](https://mlsanity.com/paper/2608.05732/) (2026)
- [Tabby: An Open Pretraining Recipe for Time Series Foundation Models](https://mlsanity.com/paper/2609.13956/) (2026)
- [Interpretability in Deep Time Series Models Demands Semantic Alignment](https://mlsanity.com/paper/2602.02239/) (2026)
- [Towards Universal Representation-Based Process Control](https://mlsanity.com/paper/2609.30790/) (2026)
- [When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events](https://mlsanity.com/paper/2609.39386/) (2026)
- [Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models](https://mlsanity.com/paper/2609.39445/) (2026)
- [Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs](https://mlsanity.com/paper/2610.02058/) (2026)
- [What Does an Observability Foundation Model Know?](https://mlsanity.com/paper/2610.05577/) (2026)

## Abstract (arXiv)

Time series foundation models (TSFMs) promise to be powerful tools for a wide range of applications. However, their internal representations and learned concepts are still not well understood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. Additionally, we explore the concepts learned by these models - such as periodicity and trends - and how these can be manipulated through latent space steering to influence model behavior. Our experiments show that steering interventions can introduce new features, e.g., adding periodicity or trends to signals that initially lacked them. These findings underscore the value of representational analysis for optimizing models and demonstrate how conceptual steering offers new possibilities for more controlled and efficient time series analysis with TSFMs.