mlsanityData to 14 Oct 2026

Are Language Models Efficient Reasoners? A Perspective from Logic Programming

arXiv 2510.25626 ↗First posted 29 Oct 2025cs.CL cs.AI cs.LGcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 53 of its 93 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

53 of 93 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?28 Mar
  2. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles16 Sep
  3. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic11 Aug
  4. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Language20 Dec
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  6. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025s1: Simple test-time scalingcited by 1,341
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
35 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  5. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  6. 2023Faithful Chain-of-Thought Reasoningcited by 288
  7. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  8. 2020Transformers as Soft Reasoners over Languagecited by 270
  9. 2024Token-Budget-Aware LLM Reasoningcited by 247
  10. 2025Training Language Models to Reason Efficientlycited by 247
  11. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  12. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 169
  13. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
  14. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  15. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  16. 2023Baldur: Whole-Proof Generation and Repair with Large Language Modelscited by 119
  17. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  18. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  19. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
  20. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  21. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
  22. 2023Strategic Reasoning with Language Modelscited by 64
  23. 2024Do language models plan ahead for future tokens?cited by 49
  24. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  25. 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Modelscited by 35
  26. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  27. 2020Critical Thinking for Language Modelscited by 31
  28. 2024Transformers Struggle to Learn to Searchcited by 19
  29. 2022Fifty Years of Prolog and Beyondcited by 15
  30. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  31. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  32. 2023World Models for Math Story Problemscited by 11
  33. 2025RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluationcited by 11
  34. 2025Causality can systematically address the monsters under the bench(marks)cited by 4
  35. 2017Foundations of Declarative Data Analysis Using Limit Datalog Programscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoningcited by 0
  2. 2026Learning to Reason Efficiently with A* Post-Trainingcited by 0
  3. 2026Scaling of Capability and Efficiency at Inference Time in Large Reasoning Modelscited by 0
  4. 2026LLMs for Executable Multi-Agent System Specification Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  2. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
  3. 2025DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoning
  4. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  5. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
  6. 2024Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning
  7. 2025EfficientXLang: Towards Improving Token Efficiency Through Cross-Lingual Reasoning
  8. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models