mlsanityData to 13 Oct 2026

Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability

arXiv 2511.05541 ↗First posted 30 Oct 2025cs.CL cs.AI cs.LGcited by 17

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in language models.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 35 of its 55 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

35 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Training9 Oct
  2. 2025Dense SAE Latents Are Features, Not Bugs18 Jun
  3. 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
  4. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing23 Feb
  6. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
  7. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features28 Jan
  8. 2024BatchTopK Sparse Autoencoders9 Dec
  9. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  10. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models31 Jul
  11. 2024Scaling and evaluating sparse autoencoders6 Jun
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 23

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,968
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,316
  8. 2020Concept Bottleneck Modelscited by 1,027
  9. 2022Toy Models of Superpositioncited by 950
  10. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 272
  11. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 140
  12. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 122
11 more
  1. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 72
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 60
  3. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  4. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 30
  5. 2025Transcoders Beat Sparse Autoencoders for Interpretabilitycited by 23
  6. 2025AI Alignment at Your Discretioncited by 20
  7. 2024Analyzing (In)Abilities of SAEs via Formal Languagescited by 16
  8. 2025From Thought to Action: How a Hierarchy of Neural Dynamics Supports Language Productioncited by 9
  9. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  10. 2016Fast Low-rank Shared Dictionary Learning for Image Classificationcited by 7
  11. 2015Beyond Low Rank + Sparse: Multi-scale Low Rank Matrix Decompositioncited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 29
  2. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 21
  3. 2026Steering at the Source: Style Modulation Heads for Robust Persona Controlcited by 5
  4. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 4
  5. 2026Tensor Product Representation Probes Reveal Shared Structure Across Linear Directionscited by 1
  6. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  7. 2026Residualized Temporal Sparse Autoencoders for Interpreting Diffusion Modelscited by 1
  8. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  9. 2026Turn-Averaged SAEs for Feature Discovery and Long-Context Attributioncited by 1
  10. 2026Laguerre Geometry for Interpreting Large Language Modelscited by 0
  11. 2026Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representationscited by 0
  12. 2026Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?cited by 0
5 more
  1. 2026Identifying Temporal Features within Transcoders for Time Sensitive Factual Recallcited by 0
  2. 2026Interference Beyond Geometry in Concept Extractioncited by 0
  3. 2026Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discoverycited by 0
  4. 2026Towards Fast and Disentangled Counterfactuals for Visual Foundation Modelscited by 0
  5. 2026ClasSAE: Class-Aligned Sparse Autoencoders via Differentiable Feature-Class Affinitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  2. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures
  3. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  4. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  5. 2024Automatically Interpreting Millions of Features in Large Language Models
  6. 2025Priors in Time: Missing Inductive Biases for Language Model Interpretability
  7. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  8. 2025Evaluating SAE interpretability without explanations