mlsanityData to 7 Oct 2026

CORE: Comprehensive Ontological Relation Evaluation for Large Language Models

arXiv 2602.06446 ↗First posted 6 Feb 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models (LLMs) perform well on many reasoning benchmarks, yet existing evaluations rarely assess their ability to distinguish between meaningful semantic relations and genuine unrelatedness. We introduce CORE (Comprehensive Ontological Relation Evaluation), a dataset of 225K multiple-choice questions spanning 74 disciplines, together with a general-domain open-source benchmark of 203 rigorously validated questions (Cohen's Kappa = 1.0) covering 24 semantic relation types with equal representation of unrelated pairs. A human baseline from 1,000+ participants achieves 92.6% accuracy (95.1% on unrelated pairs). In contrast, 29 state-of-the-art LLMs achieve 48.25-70.9% overall accuracy, with near-ceiling performance on related pairs (86.5-100%) but severe degradation on unrelated pairs (0-41.35%), despite assigning similar confidence (92-94%). Expected Calibration Error increases 2-4x on unrelated pairs, and a mean semantic collapse rate of 37.6% indicates systematic generation of spurious relations. On the CORE 225K MCQs dataset, accuracy further drops to approximately 2%, highlighting substantial challenges in domain-specific semantic reasoning. We identify unrelatedness reasoning as a critical, under-evaluated frontier for LLM evaluation and safety.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 42 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

32 of 42 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
20 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  5. 2019Attention is not Explanationcited by 916
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2025Humanity's Last Examcited by 467
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2023Augmented Language Models: a Surveycited by 398
  13. 2020Experience Grounds Languagecited by 241
  14. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  15. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  16. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  17. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  18. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  19. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 12
  20. 2022Probing for Understanding of English Verb Classes and Alternations in Large Pre-trained Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning
  2. 2026Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI
  3. 2025ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction
  4. 2025Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons
  5. 2025Reasoning of Large Language Models over Knowledge Graphs with Super-Relations
  6. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
  7. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering
  8. 2026KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models