mlsanityData to 14 Oct 2026

Finding Manifolds With Bilinear Autoencoders

arXiv 2510.16820 ↗First posted 19 Oct 2025cs.LGcited by 5

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders are a standard tool for uncovering interpretable latent representations in neural networks. Yet, their interpretation depends on the inputs, making their isolated study incomplete. Polynomials offer a solution; they serve as algebraic primitives that can be analysed without reference to input and can describe structures ranging from linear concepts to complicated manifolds. This work uses bilinear autoencoders to efficiently decompose representations into quadratic polynomials. We discuss improvements that induce importance ordering, clustering, and activation sparsity. This is an initial step toward nonlinear yet analysable latents through their algebraic properties.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 34 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

23 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025The Origins of Representation Manifolds in Large Language Models23 May
  2. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  3. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry3 Mar
  4. 2024Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs15 Oct

Other references · 19

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  7. 2016Language Modeling with Gated Convolutional Networkscited by 702
  8. 2024Scaling and evaluating sparse autoencoderscited by 553
  9. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  10. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  11. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  12. 2024BatchTopK Sparse Autoencoderscited by 130
7 more
  1. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
  2. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 34
  3. 2024Bilinear MLPs enable weight-based mechanistic interpretabilitycited by 28
  4. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  5. 2024Towards Compositional Interpretability for XAIcited by 12
  6. 2021Compositionality as we see it, everywhere around uscited by 11
  7. 2025Compositionality Unlocks Deep Interpretable Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  2. 2026The Information Geometry of Softmax: Probing and Steeringcited by 11
  3. 2026PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decodingcited by 3
  4. 2026SMIXAE: Towards Unsupervised Manifold Discovery in Language Modelscited by 1
  5. 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Bilinear autoencoders find interpretable manifolds
  2. 2025Equivariant Sparse Autoencoders: Mechanistic Interpretability of Neural Networks on Symmetric Data
  3. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
  4. 2022Benign Autoencoders
  5. 2026Finding Belief Geometries with Sparse Autoencoders
  6. 2023Quantized Fourier and Polynomial Features for more Expressive Tensor Network Models
  7. 2025Discovering Hidden Algebraic Structures via Transformers with Rank-Aware Beam GRPO
  8. 2025Sparse Autoencoders, Again?