mlsanityData to 7 Oct 2026

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models

arXiv 2609.26610 ↗First posted 22 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation also fits: Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Despite their outstanding performance on many NLP tasks, LLMs face serious challenges related to semantic abstraction. In this study, we are interested in understanding how LLMs leverage abstract semantic knowledge in natural language inference (NLI), which requires sophisticated linguistic capabilities to interpret implicit meanings, contextual conceptual relationships, and semantic connections between words and phrases. To this end, we propose a methodological framework for constructing new semantic knowledge at a higher level of abstraction, which we define under the notions of semantic compatibility and incompatibility for NLI. In this framework, the meaning of the lexical-semantic relations between the premise and the hypothesis is reconfigured to achieve a more flexible semantic network that induces different reasoning paths in LLMs. These new pathways show a consistent pattern of responses that allows agreement on a single response. The results demonstrate that our proposal allows to discover and compensate for LLMs' semantic knowledge gaps in NLI, achieving significant improvements in accuracy, exceeding 10% for some models, and in particular for the non-entailment class. It is essential to note that LLMs need structured knowledge and not just more data to bridge reasoning gaps. Our hybrid approach directs attention to overlooked word relationships, allowing models to synthesize missing information. We believe that the future lies not in increasing model size, but in creating a semantic scafolding that mimics the flexibility of human thinking. Hopefully, our proposal will enable the development of more robust agents and interpretable reasoning, guiding AI toward reliable language understanding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 46 of its 73 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

46 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models30 Jul
  2. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity24 Jul
  3. 2024Are LLMs classical or nonmonotonic reasoners? Lessons from generics5 Jun
  4. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus18 Mar
  5. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance14 Mar
  6. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs18 Feb
  7. 2023Large Language Models Are Not Strong Abstract Reasoners31 May
  8. 2022Making Large Language Models Better Reasoners with Step-Aware Verifier6 Jun

Other references · 38

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  11. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  12. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
26 more
  1. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  2. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  3. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  4. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  5. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  7. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  8. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  9. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  10. 2018Improving Natural Language Inference Using External Knowledge in the Science Questions Domaincited by 48
  11. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  12. 2023Can Large Language Models Capture Dissenting Human Voices?cited by 33
  13. 2021Dimensions of Commonsense Knowledgecited by 29
  14. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  15. 2023AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graphcited by 17
  16. 2024Abstraction-of-Thought Makes Language Models Better Reasonerscited by 11
  17. 2024Large Language Models Can Better Understand Knowledge Graphs Than We Thoughtcited by 10
  18. 2019Infusing Knowledge into the Textual Entailment Task Using Graph Convolutional Networkscited by 8
  19. 2024MSciNLI: A Diverse Benchmark for Scientific Natural Language Inferencecited by 8
  20. 2022Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Modelscited by 6
  21. 2023SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Datacited by 6
  22. 2021Decomposing Natural Logic Inferences in Neural NLIcited by 4
  23. 2024Detecting Conceptual Abstraction in LLMscited by 3
  24. 2022Lexical semantics enhanced neural word embeddingscited by 1
  25. 2024A Comprehensive Evaluation of Semantic Relation Knowledge of Pretrained Language Models and Humanscited by 1
  26. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
  2. 2024Reliable Reasoning Beyond Natural Language
  3. 2022The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems
  4. 2024Logically Consistent Language Models via Neuro-Symbolic Integration
  5. 2026Multi-Granularity Reasoning for Natural Language Inference
  6. 2025SR-LLM: Rethinking the Structured Representation in Large Language Model
  7. 2025LLM Inference Enhanced by External Knowledge: A Survey
  8. 2026Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models