Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large language models represent the same reasoning in vastly different surface forms -- English prose, Python code, mathematical notation -- yet whether they share a common internal substrate across these symbolic systems remains unknown. We introduce the TriForm Benchmark (18 concepts x 6 forms x 3 instances = 324 stimuli) and study five LLMs (1.6B-8B) across three architecture families. Using permutation-corrected RSA, cross-form probing, and activation patching, we find converging evidence for a Format-Agnostic Reasoning Subspace (FARS) in middle layers. We make FARS concrete: concept-centroid PCA extracts a 10-dimensional subspace that amplifies concept structure 3x while suppressing form information to near zero. Replacing only these 10 dimensions during cross-form patching preserves 90-96% of model output -- far exceeding both full activation replacement (44-56%) and variance-maximizing PCA (60-74%) -- while ablating them causes targeted disruption. FARS generalizes to held-out concepts and converges across architectures (CCA > 0.79 for all model pairs), providing within-modality evidence for the Platonic Representation Hypothesis. We further discover a declarative-procedural asymmetry: representations are far more compatible between prose and mathematics than between either and code, suggesting that the critical axis of divergence is not linguistic vs. formal but declarative vs. procedural.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 14 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
10 of 14 references.
Closest ancestors · 3
Same direction, newest first
- 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Language11 Jun
- 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers13 Nov
- 2024Large Language Models Are Cross-Lingual Knowledge-Free Reasoners24 Jun
Other references · 7
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
- 2024How do Large Language Models Handle Multilingualism?cited by 148
- 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
- 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
- 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Logical subspace in LLMscited by 0
- 2026Logical subspace in LLMs26 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning
- 2026Unveiling the Reasoning Process of Large Language Models
- 2026Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
- 2024Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning
- 2024Calibrating Reasoning in Language Models with Internal Consistency
- 2026Discovering a Shared Logical Subspace: Steering LLM Logical Reasoning via Alignment of Natural-Language and Symbolic Views
- 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models