mlsanityData to 14 Oct 2026

TREAT: Evaluating Access to Formal Knowledge across Equivalent Mathematical Representations

arXiv 2608.07540 ↗First posted 29 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

AI systems increasingly operate between flexible input representations and formal objects used by downstream tools. A key challenge is recognizing when an unfamiliar formulation denotes a known formal object. We study this challenge through theorem recognition: given an equivalence-preserving transformation of a theorem condition, a model must recover the theorem identity associated with the standard statement. We introduce TREAT, a benchmark for evaluating whether large language models can recover known theorem identities from equivalence-preserving formula-level transformations. Rather than paraphrasing theorem text, TREAT changes the mathematical form of theorem conditions themselves, expressing known results through residual equations, witness statements, optimization identities, set relations, operator forms, and proof-intermediate characterizations. Starting from scraped theorem pages, we filter for entries with usable mathematical expression forms, extract canonical theorem conditions, and generate transformed variants with recorded assumptions and inverse mappings. The final corpus contains 737 theorem identities and 29,480 transformed rows. On a test panel, the best model retrieves the correct theorem identity in only 60.73% of cases. Other systems reveal different failure modes, including abstention, wrong detection, and malformed outputs. These suggest that theorem knowledge can be fragile under equivalent changes in representation. TREAT therefore provides a controlled testbed for evaluating representation-robust access to formal knowledge, with broader relevance to domains that require stable target objects, explicit equivalence relations, validation procedures, and auditable scoring.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

13 of 21 references.

References · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  5. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  6. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  7. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  8. 2022Autoformalization with Large Language Modelscited by 263
  9. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  10. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  11. 2016DeepMath - Deep Sequence Models for Premise Selectioncited by 81
  12. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
1 more
  1. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problemscited by 8

Built on by

Not cited yet.