# Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching

arXiv: https://arxiv.org/abs/2610.19636 · HTML: https://mlsanity.com/paper/2610.19636/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics](https://mlsanity.com/paper/2109.00110/) (2021)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [Learning to Plan Chemical Syntheses](https://mlsanity.com/paper/1708.04202/) (2017)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [The Lean mathematical library](https://mlsanity.com/paper/1910.09336/) (2019)
- [A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs](https://mlsanity.com/paper/2003.07743/) (2020)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Thinking Fast and Slow in AI](https://mlsanity.com/paper/2010.06002/) (2020)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence](https://mlsanity.com/paper/2206.08474/) (2022)
- [Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence](https://mlsanity.com/paper/2306.02572/) (2023)

## Abstract (arXiv)

Large language models (LLMs) hold broad knowledge in one set of weights, but lack interpretability and trustworthy reasoning, especially in high-stakes use cases. A neuro-symbolic model, with its interpretable architecture and System 2 reasoning capacity, is a promising alternative if built at the scale of LLMs; however, its symbolic backbone is usually constructed with human effort, so existing neuro-symbolic models are bottlenecked by the scope of the symbol component and usually domain-specific. A data-driven method could instead build a larger symbolic system by aligning and merging existing ones. We target its first, foundational step: matching units across these heterogeneous systems. Our method, SAE-IDF, encodes a symbol in its native context into a sparse code with a sparse autoencoder (SAE), re-weights each dimension by its rarity and matches across systems. On 12 tasks across 6 kinds of symbolic system, SAE-IDF outperforms the same LLM's dense baseline on all tasks with Gemma-2-9B and all but one with Llama-3.1-8B, by up to 45 points, consistently across SAE layers and configurations. SAE-IDF is also interpretable: every dimension of the sparse code has a human-readable meaning, so each match decomposes into named concepts (features). Our work paves the way to a universal symbolic backbone.