mlsanityData to 7 Oct 2026

Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models

arXiv 2410.01280 ↗First posted 2 Oct 2024cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In-context learning, the ability to adapt based on a few examples in the input prompt, is a ubiquitous feature of large language models (LLMs). However, as LLMs' in-context learning abilities continue to improve, understanding this phenomenon mechanistically becomes increasingly important. In particular, it is not well-understood how LLMs learn to solve specific classes of problems, such as reinforcement learning (RL) problems, in-context. Through three different tasks, we first show that Llama 3 70B can solve simple RL problems in-context. We then analyze the residual stream of Llama using Sparse Autoencoders (SAEs) and find representations that closely match temporal difference (TD) errors. Notably, these representations emerge despite the model only being trained to predict the next token. We verify that these representations are indeed causally involved in the computation of TD errors and Q-values by performing carefully designed interventions on them. Taken together, our work establishes a methodology for studying and manipulating in-context learning with SAEs, paving the way for a more mechanistic understanding.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 34 of its 73 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

34 of 73 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2022In-context Learning and Induction Headscited by 930
22 more
  1. 2022Toy Models of Superpositioncited by 928
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  5. 2024Scaling and evaluating sparse autoencoderscited by 530
  6. 2022Using cognitive psychology to understand GPT-3cited by 389
  7. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  8. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  13. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  14. 2024How to use and interpret activation patchingcited by 191
  15. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  16. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  17. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  18. 2016Deep Successor Reinforcement Learningcited by 72
  19. 2023Meta-in-context learning in large language modelscited by 59
  20. 2024In-context learning agents are asymmetric belief updaterscited by 13
  21. 2023Reinforcement Learning with Simple Sequence Priorscited by 11
  22. 2024Large Language Models are Biased Reinforcement Learnerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  2. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 24
  3. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencodingcited by 19
  4. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  5. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learningcited by 10
  6. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerabilitycited by 8
  7. 2025Large Language Models Think Too Fast To Explore Effectivelycited by 7
  8. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  9. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  10. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 2
  11. 2026Hierarchical Experimentalist Agentscited by 2
  12. 2025A circuit for predicting hierarchical structure in-context in Large Language Modelscited by 1
3 more
  1. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  2. 2026Generalizing Linear Autoencoder Recommenders with Decoupled Expected Quadratic Losscited by 0
  3. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0