mlsanityData to 7 Oct 2026

Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models

arXiv 2407.08440 ↗First posted 11 Jul 2024cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Although Large Language Models (LLMs) have demonstrated strong ability, they are further supposed to be controlled and guided by in real-world scenarios to be safe, accurate, and intelligent. This demands the possession of capability of LLMs. However, no prior work has made a clear evaluation of the inferential rule-following capability of LLMs. Previous studies that try to evaluate the inferential rule-following capability of LLMs fail to distinguish the inferential rule-following scenarios from the instruction-following scenarios. Therefore, this paper first clarifies the concept of inferential rule-following and proposes a comprehensive benchmark, RuleBench, to evaluate a diversified range of inferential rule-following abilities. Our experimental results on a variety of LLMs show that they are still limited in following rules. Our analysis based on the evaluation results provides insights into the improvements for LLMs toward a better inferential rule-following intelligent agent. We further propose Inferential Rule-Following Tuning (IRFT). The experimental results show that through IRFT, LLMs can learn abstract rule-following abilities from purely synthetic data and then generalize to RuleBench. The data and code can be found at: https://anonymous.4open.science/r/llm-rule-following-B3E3/

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 17 of its 40 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

17 of 40 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  10. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  11. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  12. 2023Can LLMs Follow Simple Rules?cited by 45
5 more
  1. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  2. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  3. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  4. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  5. 2018Overview of CAIL2018: Legal Judgment Prediction Competitioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 26
  2. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  3. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 21
  4. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  5. 2024RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answeringcited by 13
  6. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  7. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5
  8. 2025LLMs Lean on Priors, Not Programming Language Semanticscited by 5
  9. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
  10. 2025Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimatecited by 3
  11. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  12. 2025Semi-structured LLM Reasoners Can Be Rigorously Auditedcited by 2
11 more
  1. 2025GENIE-ASI: Generative Instruction and Executable Code for Analog Subcircuit Identificationcited by 2
  2. 2026Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steeringcited by 2
  3. 2026Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Modelscited by 2
  4. 2025From Literal to Liberal: A Meta-Prompting Framework for Eliciting Human-Aligned Exception Handling in Large Language Modelscited by 1
  5. 2026Verification of the Implicit World Model in a Generative Model via Adversarial Sequencescited by 1
  6. 2026What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Modelscited by 1
  7. 2025Shuttle Between the Instructions and the Parameters of Large Language Modelscited by 0
  8. 2025CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculationcited by 0
  9. 2026Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilitiescited by 0
  10. 2026LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Taskscited by 0
  11. 2026Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderationcited by 0