mlsanityData to 7 Oct 2026

AIMO Interpretability Challenge

arXiv 2607.13899 ↗First posted 15 Jul 2026cs.AIcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We propose the AIMO Interpretability Challenge, a competition on distinguishing robust from spurious reasoning in frontier mathematical language models based on the models' internal mechanisms. The challenge is motivated by a central limitation of standard reasoning benchmarks: strong final-answer accuracy does not reveal whether a model relies on stable reasoning mechanisms or exploits brittle reasoning shortcuts. Building on AI Mathematical Olympiad (AIMO) problems and submissions, together with resources from the Fields Model Initiative, the competition will provide (1) newly-published olympiad-level math reasoning problems and their symbolic representations, allowing generation of novel functional variants, (2) access to frontier reasoning models, and (3) assessments of models' adversarial robustness on these problems. Participants will use these resources, along with our computing infrastructure support, to develop methods for identifying which models solve problems robustly. Our competition will also create a new, open robustness benchmark and baseline systems, aiming to provide a lasting foundation for standard benchmarking in mathematical reasoning and interpretability. Scientifically, the competition connects interpretability and generalization research around a central question in AI research: can we determine if, and to what extent, the decision-making of frontier AI models is generalizable and thus, reliable?

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 23 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

17 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct

Other references · 16

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
  5. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  6. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  7. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  8. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  9. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  10. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  11. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  12. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
4 more
  1. 2025Can Interpretation Predict Behavior on Unseen Data?cited by 8
  2. 2025Attend or Perish: Benchmarking Attention in Algorithmic Reasoningcited by 2
  3. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scalingcited by 2
  4. 2026Tracing Uncertainty in Language Model "Reasoning"cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
  2. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
  3. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
  4. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  5. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models
  6. 2026Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning
  7. 2025AI Mathematician: Towards Fully Automated Frontier Mathematical Research
  8. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems