mlsanityData to 7 Oct 2026

ACCORD: Closing the Commonsense Measurability Gap

arXiv 2406.02804 ↗First posted 4 Jun 2024cs.AI cs.CL cs.LGcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present ACCORD, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. Benchmarking state-of-the-art LLMs -- including GPT-4o (2024-05-13), Llama-3-70B-Instruct, and Mixtral-8x22B-Instruct-v0.1 -- shows performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 80 of its 110 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

80 of 110 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text15 Jul
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  3. 2023Probing Physical Reasoning with Counter-Commonsense Context4 Jun
  4. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May
  5. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness21 Apr
  6. 2023Augmented Language Models: a Survey15 Feb
  7. 2023Benchmarks for Automated Commonsense Reasoning: A Survey9 Feb
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb
  9. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  10. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  11. 2022Language models show human-like content effects on reasoning tasks14 Jul

Other references · 69

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Mistral 7Bcited by 3,065
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  12. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  13. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  14. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  15. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  16. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  17. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  18. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  19. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  20. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  21. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  22. 2022Large Language Models with Controllable Working Memorycited by 169
  23. 2021Explaining Answers with Entailment Treescited by 148
  24. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  25. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  26. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  27. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  28. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  29. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  30. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  31. 2019Counterfactual Story Reasoning and Generationcited by 87
  32. 2023Context-faithful Prompting for Large Language Modelscited by 76
  33. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  34. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  35. 2020Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoningcited by 65
  36. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 64
  37. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  38. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025mSCoRe: a Multilingual and Scalable Benchmark for Skill-based Commonsense Reasoning
  2. 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
  3. 2026QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
  4. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
  5. 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models
  6. 2025Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
  7. 2025seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs
  8. 2026Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregation