mlsanityData to 7 Oct 2026

AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph

arXiv 2311.09174 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge. While existing resources only touch nouns or verbs within simplified events or specific domains, AbsPyramid collects abstract knowledge for three components of diverse events to comprehensively evaluate the abstraction ability of language models in the open domain. Experimental results demonstrate that current LLMs face challenges comprehending abstraction knowledge in zero-shot and few-shot settings. By training on our rich abstraction knowledge, we find LLMs can acquire basic abstraction abilities and generalize to unseen events. In the meantime, we empirically show that our benchmark is comprehensive to enhance LLMs across two previous abstraction tasks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 39 of its 80 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

39 of 80 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
27 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  8. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  11. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  12. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  13. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  14. 2021Explaining Answers with Entailment Treescited by 148
  15. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89
  16. 2019GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledgecited by 78
  17. 2022PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognitioncited by 29
  18. 2021ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualitiescited by 28
  19. 2023CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoningcited by 25
  20. 2019Probabilistic Models of Relational Implicationcited by 17
  21. 2022Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualizationcited by 17
  22. 2022Entailment Graph Learning with Textual Entailment and Soft Transitivitycited by 11
  23. 2021Multivalent Entailment Graphs for Question Answeringcited by 10
  24. 2022Smoothing Entailment Graphs with Language Modelscited by 8
  25. 2021Incorporating Temporal Information in Entailment Graph Miningcited by 7
  26. 2018Representing Verbs as Argument Conceptscited by 6
  27. 2022Cross-lingual Inference with A Chinese Entailment Graphcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 1; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  2. 2024CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoningcited by 28
  3. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  4. 2025AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corporacited by 23
  5. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  6. 2024CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecastingcited by 19
  7. 2024AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimationcited by 14
  8. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  9. 2024Abstraction-of-Thought Makes Language Models Better Reasonerscited by 11
  10. 2024ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertaintycited by 8
  11. 2024Intention Knowledge Graph Construction for User Intention Relation Modelingcited by 6
  12. 2024Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstractioncited by 4
5 more
  1. 2024On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directionscited by 3
  2. 2024KNOWCOMP POKEMON Team at DialAM-2024: A Two-Stage Pipeline for Detecting Relations in Dialogical Argument Miningcited by 2
  3. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  4. 2026Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Wordscited by 0
  5. 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Modelscited by 0