# pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference

arXiv: https://arxiv.org/abs/1810.08854 · HTML: https://mlsanity.com/paper/1810.08854/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2018.

## Built on by (most cited first, 10 of 10)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Do Question Answering Modeling Improvements Hold Across Benchmarks?](https://mlsanity.com/paper/2102.01065/) (2021)
- [Distilling Relation Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2110.15705/) (2021)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks](https://mlsanity.com/paper/2403.09207/) (2024)
- [Fine-grained Contrastive Learning for Definition Generation](https://mlsanity.com/paper/2210.00543/) (2022)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)
- [Learning to Borrow -- Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion](https://mlsanity.com/paper/2204.13097/) (2022)
- [How can Deep Learning Retrieve the Write-Missing Additional Diagnosis from Chinese Electronic Medical Record For DRG](https://mlsanity.com/paper/2303.16757/) (2023)

## Abstract (arXiv)

Reasoning about implied relationships (e.g., paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems. This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships. Our pairwise embeddings are computed as a compositional function on word representations, which is learned by maximizing the pointwise mutual information (PMI) with the contexts in which the two words co-occur. We add these representations to the cross-sentence attention layer of existing inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or replacing existing word embeddings. Experiments show a gain of 2.7% on the recently released SQuAD2.0 and 1.3% on MultiNLI. Our representations also aid in better generalization with gains of around 6-7% on adversarial SQuAD datasets, and 8.8% on the adversarial entailment test set by Glockner et al. (2018).