# Smoothing Entailment Graphs with Language Models

arXiv: https://arxiv.org/abs/2208.00318 · HTML: https://mlsanity.com/paper/2208.00318/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 30 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [Probabilistic Models of Relational Implication](https://mlsanity.com/paper/1907.12048/) (2019)
- [Do Neural Nets Learn Statistical Laws behind Natural Language?](https://mlsanity.com/paper/1707.04848/) (2017)
- [Entailment Graph Learning with Textual Entailment and Soft Transitivity](https://mlsanity.com/paper/2204.03286/) (2022)
- [Multivalent Entailment Graphs for Question Answering](https://mlsanity.com/paper/2104.07846/) (2021)
- [Language Models for Lexical Inference in Context](https://mlsanity.com/paper/2102.05331/) (2021)
- [Incorporating Temporal Information in Entailment Graph Mining](https://mlsanity.com/paper/2109.09412/) (2021)
- [Modality and Negation in Event Extraction](https://mlsanity.com/paper/2109.09393/) (2021)
- [Distributional Inclusion Hypothesis for Tensor-based Composition](https://mlsanity.com/paper/1610.04416/) (2016)
- [Cross-lingual Inference with A Chinese Entailment Graph](https://mlsanity.com/paper/2203.06264/) (2022)
- [Blindness to Modality Helps Entailment Graph Mining](https://mlsanity.com/paper/2109.10227/) (2021)

## Built on by (most cited first, 8 of 8)

- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph](https://mlsanity.com/paper/2311.09174/) (2023)
- [AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation](https://mlsanity.com/paper/2402.10646/) (2024)
- [TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks](https://mlsanity.com/paper/2403.09207/) (2024)
- [The Case for Scalable, Data-Driven Theory: A Paradigm for Scientific Progress in NLP](https://mlsanity.com/paper/2312.00349/) (2023)
- [From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation](https://mlsanity.com/paper/2306.04170/) (2023)
- [Explicit Inductive Inference using Large Language Models](https://mlsanity.com/paper/2408.14467/) (2024)
- [LLMs are Frequency Pattern Learners in Natural Language Inference](https://mlsanity.com/paper/2505.21011/) (2025)

## Abstract (arXiv)

The diversity and Zipfian frequency distribution of natural language predicates in corpora leads to sparsity in Entailment Graphs (EGs) built by Open Relation Extraction (ORE). EGs are computationally efficient and explainable models of natural language inference, but as symbolic models, they fail if a novel premise or hypothesis vertex is missing at test-time. We present theory and methodology for overcoming such sparsity in symbolic models. First, we introduce a theory of optimal smoothing of EGs by constructing transitive chains. We then demonstrate an efficient, open-domain, and unsupervised smoothing method using an off-the-shelf Language Model to find approximations of missing premise predicates. This improves recall by 25.1 and 16.3 percentage points on two difficult directional entailment datasets, while raising average precision and maintaining model explainability. Further, in a QA task we show that EG smoothing is most useful for answering questions with lesser supporting text, where missing premise predicates are more costly. Finally, controlled experiments with WordNet confirm our theory and show that hypothesis smoothing is difficult, but possible in principle.