mlsanityData to 13 Oct 2026

CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

arXiv 1908.06177 ↗First posted 16 Aug 2019cs.LG cs.CL cs.LOcited by 158

Not on the map. Why?

Cited by 83 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent success of natural language understanding (NLU) systems has been troubled by results highlighting the failure of these models to generalize in a systematic and robust way. In this work, we introduce a diagnostic benchmark suite, named CLUTRR, to clarify some key issues related to the robustness and systematicity of NLU systems. Motivated by classic work on inductive logic programming, CLUTRR requires that an NLU system infer kinship relations between characters in short stories. Successful performance on this task requires both extracting relationships between entities, as well as inferring the logical rules governing these relationships. CLUTRR allows us to precisely measure a model's ability for systematic generalization by evaluating on held-out combinations of logical rules, and it allows us to evaluate a model's robustness by adding curated noise facts. Our empirical results highlight a substantial performance gap between state-of-the-art NLU models (e.g., BERT and MAC) and a graph neural network model that works directly with symbolic inputs---with the graph-based model exhibiting both stronger generalization and greater robustness.

Key facts. Cited by 158 papers in the map (data through 13 Oct 2026). First posted on arXiv 16 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 21; 2023: 31; 2024: 27; 2025: 32; 2026: 32.

158 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 829
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 322
  4. 2023Faithful Chain-of-Thought Reasoningcited by 286
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  7. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  8. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  9. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  11. 2022On the Paradox of Learning to Reason from Datacited by 93
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
38 more
  1. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  2. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  3. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 74
  4. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  5. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  6. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
  7. 2023Large Language Models can Learn Rulescited by 67
  8. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  9. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
  10. 2023Natural Language Reasoning, A Surveycited by 61
  11. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  12. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  13. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  14. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  15. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  16. 2021Systematic Generalization with Edge Transformerscited by 38
  17. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  18. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  19. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  20. 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
  21. 2022Language Models as Inductive Reasonerscited by 32
  22. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  23. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
  24. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  25. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  26. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 27
  27. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  28. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  29. 2024Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Modelscited by 23
  30. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  31. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoningcited by 22
  32. 2025Parallel Continuous Chain-of-Thought with Jacobi Iterationcited by 22
  33. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approachcited by 21
  34. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Modelscited by 21
  35. 2023Neural Compositional Rule Learning for Knowledge Graph Reasoningcited by 20
  36. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  37. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Modelscited by 19
  38. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19

50 most cited of 158 All 158 (JSON)