mlsanityData to 14 Oct 2026

Disentangling concept semantics via multilingual averaging in Sparse Autoencoders

arXiv 2508.14275 ↗First posted 19 Aug 2025cs.CL cs.AIcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Connecting LLMs with formal knowledge representation and reasoning is a promising approach to address their shortcomings. Embeddings and sparse autoencoders are widely used to represent textual content, but the semantics are entangled with syntactic and language-specific information. We propose a method that isolates concept semantics in Large Langue Models by averaging concept activations derived via Sparse Autoencoders. We create English text representations from OWL ontology classes, translate the English into French and Chinese and then pass these texts as prompts to the Gemma 2B LLM. Using the open source Gemma Scope suite of Sparse Autoencoders, we obtain concept activations for each class and language version. We average the different language activations to derive a conceptual average. We then correlate the conceptual averages with a ground truth mapping between ontology classes. Our results give a strong indication that the conceptual average aligns to the true relationship between classes when compared with a single language by itself. The result hints at a new technique which enables mechanistic interpretation of internal network states with higher accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 9 of its 17 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

9 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models7 Mar
  2. 2024Automatically Interpreting Millions of Features in Large Language Models17 Oct
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep

Other references · 6

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  2. 2023Language Models Represent Space and Timecited by 316
  3. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  4. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 32
  5. 2023Agent-OM: Leveraging LLM Agents for Ontology Matchingcited by 27
  6. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AlignSAE: Concept-Aligned Sparse Autoencoders
  2. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  3. 2026Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features
  4. 2026Navigating the Concept Space of Language Models
  5. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs
  6. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  7. 2025OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
  8. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit