mlsanityData to 7 Oct 2026

Exploring Representations and Interventions in Time Series Foundation Models

arXiv 2409.12915 ↗First posted 19 Sep 2024cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Time series foundation models (TSFMs) promise to be powerful tools for a wide range of applications. However, their internal representations and learned concepts are still not well understood. In this study, we investigate the structure and redundancy of representations across various TSFMs, examining the self-similarity of model layers within and across different model sizes. This analysis reveals block-like redundancy in the representations, which can be utilized for informed pruning to improve inference speed and efficiency. Additionally, we explore the concepts learned by these models - such as periodicity and trends - and how these can be manipulated through latent space steering to influence model behavior. Our experiments show that steering interventions can introduce new features, e.g., adding periodicity or trends to signals that initially lacked them. These findings underscore the value of representational analysis for optimizing models and demonstrate how conceptual steering offers new possibilities for more controlled and efficient time series analysis with TSFMs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

25 of 40 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2022Toy Models of Superpositioncited by 928
  9. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  10. 2024Chronos: Learning the Language of Time Seriescited by 712
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  12. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
13 more
  1. 2023A decoder-only foundation model for time-series forecastingcited by 650
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  3. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  4. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  5. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  6. 2021Monash Time Series Forecasting Archivecited by 256
  7. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  8. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  9. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  10. 2024TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysiscited by 37
  11. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29
  12. 2022Weak Supervision for Affordable Modeling of Electrocardiogram Datacited by 8
  13. 2024Implicit Reasoning in Deep Time Series Forecastingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  2. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 5
  3. 2025Investigating Compositional Reasoning in Time Series Foundation Modelscited by 3
  4. 2025time2time: Causal Intervention in Hidden States to Simulate Rare Events in Time Series Foundation Modelscited by 3
  5. 2025The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Modelscited by 3
  6. 2025On the Internal Semantics of Time-Series Foundation Modelscited by 3
  7. 2026Universal Redundancies in Time Series Foundation Modelscited by 3
  8. 2025Time-Prompt: Integrated Heterogeneous Prompts for Unlocking LLMs in Time Series Forecastingcited by 1
  9. 2025Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detectioncited by 1
  10. 2026TabPFN Through The Looking Glass: An interpretability study of TabPFN and its internal representationscited by 1
  11. 2026Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Functioncited by 1
  12. 2026CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuitscited by 1
7 more
  1. 2026Tabby: An Open Pretraining Recipe for Time Series Foundation Modelscited by 1
  2. 2026Interpretability in Deep Time Series Models Demands Semantic Alignmentcited by 0
  3. 2026EIDOS: Latent-Space Predictive Learning for Time Series Foundation Modelscited by 0
  4. 2026Towards Universal Representation-Based Process Controlcited by 0
  5. 2026When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Eventscited by 0
  6. 2026Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Modelscited by 0
  7. 2026Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMscited by 0