mlsanityData to 14 Oct 2026

Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?

arXiv 2504.00509 ↗First posted 1 Apr 2025cs.AI cs.CL cs.LGcited by 15

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The rapid escalation from elementary school-level to frontier problems of the difficulty for LLM benchmarks in recent years have weaved a miracle for researchers that we are only inches away from surpassing human intelligence. However, is the LLMs' remarkable reasoning ability indeed comes from true intelligence by human standards, or are they simply reciting solutions witnessed during training at an Internet level? To study this problem, we propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's recitation behavior when asked simple reasoning problems but with conditions subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly, we found existing cutting-edge LLMs unanimously exhibits extremely severe recitation behavior; by changing one phrase in the condition, top models such as OpenAI-o1 and DeepSeek-R1 can suffer 60 percent performance loss on elementary school-level arithmetic and reasoning problems. Such findings are a wake-up call to the LLM community that compels us to re-evaluate the true intelligence level of cutting-edge LLMs.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 75 of its 98 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

75 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations10 Feb
  2. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI7 Nov
  3. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  4. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning5 May

Other references · 71

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  11. 2025Qwen2.5-VL Technical Reportcited by 5,587
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  13. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  14. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  15. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  16. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  17. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  18. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  19. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  20. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2025Humanity's Last Examcited by 491
  23. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  24. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  25. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  26. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  27. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
  28. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  29. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  30. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  31. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  32. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  33. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  34. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  35. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  36. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  37. 2025BIG-Bench Extra Hardcited by 125
  38. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  2. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  3. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  4. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 14
  5. 2025Exploring the System 1 Thinking Capability of Large Reasoning Modelscited by 10
  6. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  7. 2025Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosiscited by 4
  8. 2025Training-free LLM Verification via Recycling Few-shot Examplescited by 4
  9. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  10. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Modelscited by 3
  11. 2025Evaluating Visual Mathematics in Multimodal LLMs: A Multilingual Benchmark Based on the Kangaroo Testscited by 1
  12. 2025Adaptive Multi-Agent Reasoning via Automated Workflow Generationcited by 1
3 more
  1. 2026Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025cited by 1
  2. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMscited by 0
  3. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  3. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
  4. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  5. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  6. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  7. 2025Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance
  8. 2024LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems