mlsanityData to 7 Oct 2026

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

arXiv 2408.10920 ↗First posted 20 Aug 2024cs.LG cs.AI cs.NEcited by 45

Not on the map. Why?

Abstract (arXiv)

The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

27 of 46 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
15 more
  1. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2021Causal Abstractions of Neural Networkscited by 401
  4. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  5. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  7. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  8. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  9. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  10. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  11. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  12. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  13. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  14. 2023Discovering Variable Binding Circuitry with Desideratacited by 32
  15. 2023A Geometric Notion of Causal Probingcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 18.

45 papers.

Sort:

By area:

  1. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  3. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  4. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  5. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  6. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  7. 2024ICLR: In-Context Learning of Representationscited by 49
  8. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  9. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 42
  10. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  11. 2024Mechanistic?cited by 25
  12. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
33 more
  1. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  2. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  4. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  5. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  6. 2026Memory Caching: RNNs with Growing Memorycited by 12
  7. 2024Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformerscited by 11
  8. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  9. 2026How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?cited by 11
  10. 2025How Causal Abstraction Underpins Computational Explanationcited by 10
  11. 2025HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworkscited by 8
  12. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 8
  13. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  14. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 7
  15. 2025Low-Rank Adapting Models for Sparse Autoencoderscited by 6
  16. 2025Addressing divergent representations from causal interventions on neural networkscited by 6
  17. 2025Emergent Symbol-like Number Variables in Artificial Neural Networkscited by 5
  18. 2026Curveball Steering: The Right Direction To Steer Isn't Always Linearcited by 5
  19. 2025Constructing Interpretable Features from Compositional Neuron Groupscited by 3
  20. 2026PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decodingcited by 3
  21. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  22. 2026Predicting Where Steering Vectors Succeedcited by 3
  23. 2025The quasi-semantic competence of LLMs: a case study on the part-whole relationcited by 1
  24. 2026Convergent World Representations and Divergent Taskscited by 1
  25. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  26. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  27. 2025AR-KAN: Autoregressive-Weight-Enhanced Kolmogorov-Arnold Network for Time Series Forecastingcited by 0
  28. 2026Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Datacited by 0
  29. 2026Fast & Faithful Function Vectorscited by 0
  30. 2026A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretabilitycited by 0
  31. 2026MAxBench: A Multinomial Concept Recovery Benchmarkcited by 0
  32. 2026A Survey on the Linear Representation Hypothesiscited by 0
  33. 2026Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?cited by 0