Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
Reordering a set of mathematical rules without changing its meaning should preserve the correct answer, but must a model's internal representations stay invariant too? We investigate this question using synthetic multi-step function-composition problems, each presented under multiple rule orderings with the same correct answer. We measure accuracy and permutation signal-to-noise ratio (SNR), which quantifies how distinctly ordering patterns are represented relative to variation across problem instances. Across 16 language models ranging from 1B to 8B parameters, we find a pattern: models that solve reordered problems more accurately represent different rule orderings more distinctly. Layer-averaged permutation SNR is positively rank-correlated with accuracy in every synthetic setting we evaluate, with Spearman correlations reaching 0.86. These findings highlight a distinction between answer invariance and representation invariance: successful mathematical rule composition can accompany distinct internal representations between equivalent rule orderings. This motivates distinguishing answer invariance from representation invariance, and offers a representational perspective on mathematical reasoning beyond answer accuracy alone.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 19 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
12 of 19 references.
Closest ancestors · 2
Same direction, newest first
- 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
- 2024Premise Order Matters in Reasoning with Large Language Models14 Feb
Other references · 10
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
- 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
- 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
- 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
- 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
- 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
- 2025Investigating the Robustness of Deductive Reasoning with Large Language Modelscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Representation Robustness Under Executable Reasoning Constraints in Large Language Models for Mathematical Problem Solving
- 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
- 2025Examining False Positives under Inference Scaling for Mathematical Reasoning
- 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
- 2026Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
- 2025Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
- 2026Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning
- 2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity