mlsanityData to 7 Oct 2026

High-probability guarantees for linear accessibility in feature superposition

arXiv 2609.09556 ↗First posted 9 Sep 2026stat.ML cs.AI cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=Ovarepsilon(k log m)) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

22 of 42 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2022Toy Models of Superpositioncited by 928
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  7. 2024Scaling and evaluating sparse autoencoderscited by 530
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  9. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  10. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  11. 2025Language Models Use Trigonometry to Do Additioncited by 68
  12. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
10 more
  1. 2020A Theoretical Perspective on Hyperdimensional Computingcited by 43
  2. 2026When Does LeJEPA Learn a World Model?cited by 27
  3. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 24
  4. 2026How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?cited by 11
  5. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 9
  6. 2025Adversarial Attacks Leverage Interference Between Features in Superpositioncited by 7
  7. 2024On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recoverycited by 6
  8. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 5
  9. 2020Linear and Fisher Separability of Random Points in the d-dimensional Spherical Layercited by 2
  10. 2026The Scaling Properties of Implicit Deductive Reasoning in Transformerscited by 1

Built on by

Not cited yet.