mlsanityData to 7 Oct 2026

RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios

arXiv 2412.08972 ↗First posted 12 Dec 2024cs.CL cs.AIcited by 26

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper introduces RuleArena, a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions that demand long-context understanding, logical reasoning, and accurate mathematical computation. Two key attributes distinguish RuleArena from traditional rule-based reasoning benchmarks: (1) it extends beyond standard first-order logic representations, and (2) it is grounded in authentic, practical scenarios, providing insights into the suitability and reliability of LLMs for real-world applications. Our findings reveal several notable limitations in LLMs: (1) they struggle to identify and apply the appropriate rules, frequently becoming confused by similar but distinct regulations, (2) they cannot consistently perform accurate mathematical computations, even when they correctly identify the relevant rules, and (3) in general, they perform poorly in the benchmark. We also observe a significant performance boost when LLMs are provided with external tools for oracle math and logic operations. These results highlight significant challenges and promising research directions in advancing LLMs' rule-guided reasoning capabilities in real-life applications. Our codes and data are publicly available on https://github.com/skyriver-2000/RuleArena.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 43 of its 52 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

43 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Application24 Aug
  2. 2024Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities4 Jun
  3. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes22 Dec
  4. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks29 Sep
  5. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  6. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  7. 2023Specializing Smaller Language Models towards Multi-Step Reasoning30 Jan
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  12. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
23 more
  1. 2022A Survey on In-context Learningcited by 717
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  4. 2023Larger language models do in-context learning differentlycited by 386
  5. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  6. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  7. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  8. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  9. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  10. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  11. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  12. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  13. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  14. 2019Neural Module Networks for Reasoning over Textcited by 65
  15. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 49
  16. 2023Can LLMs Follow Simple Rules?cited by 45
  17. 2024Fine-tuning and Utilization Methods of Domain-specific LLMscited by 35
  18. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  19. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructionscited by 26
  20. 2024TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decisioncited by 23
  21. 2024Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Modelscited by 23
  22. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  23. 2024Supervised Chain of Thoughtcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 15.

26 papers.

Sort:

By area:

  1. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  2. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 21
  3. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  4. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 12
  5. 2025Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Provingcited by 8
  6. 2025MuSLR: Multimodal Symbolic Logical Reasoningcited by 7
  7. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  8. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  9. 2025LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Modelscited by 2
  10. 2025PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Taskscited by 2
  11. 2026SIEVE: Sample-Efficient Parametric Learning from Natural Languagecited by 2
  12. 2026Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Modelscited by 2
14 more
  1. 2025Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacycited by 1
  2. 2026Kinship Data Benchmark for Multi-hop Reasoningcited by 1
  3. 2026Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesiscited by 1
  4. 2026Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguitycited by 1
  5. 2026GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Normscited by 1
  6. 2026Context Memorization for Efficient Long Context Generationcited by 0
  7. 2026Learning to Construct Practical Agentic Systemscited by 0
  8. 2026Inducing Reasoning Primitives from Agent Tracescited by 0
  9. 2026Grounded Normative Rule Generation with Structured Searchcited by 0
  10. 2026Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Frameworkcited by 0
  11. 2026RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Modelscited by 0
  12. 2026X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalizationcited by 0
  13. 2026Learning to Prove, Not Just to Answer: Reinforcement Learning from Formal Verification for Natural-Language Logical Reasoningcited by 0
  14. 2026Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  2. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
  3. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs
  4. 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs
  5. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  6. 2024Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
  7. 2024Are Your LLMs Capable of Stable Reasoning?
  8. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs