# Lexical semantics enhanced neural word embeddings

arXiv: https://arxiv.org/abs/2210.00754 · HTML: https://mlsanity.com/paper/2210.00754/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 75 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Leveraging Knowledge Bases in LSTMs for Improving Machine Reading](https://mlsanity.com/paper/1902.09091/) (2019)
- [HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment](https://mlsanity.com/paper/1608.02117/) (2016)
- [Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints](https://mlsanity.com/paper/1706.00374/) (2017)
- [Centroid-based deep metric learning for speaker recognition](https://mlsanity.com/paper/1902.02375/) (2019)
- [Hierarchical Embeddings for Hypernymy Detection and Directionality](https://mlsanity.com/paper/1707.07273/) (2017)
- [Specialising Word Vectors for Lexical Entailment](https://mlsanity.com/paper/1710.06371/) (2017)
- [Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models](https://mlsanity.com/paper/1808.09308/) (2018)
- [Incorporating Both Distributional and Relational Semantics in Word Representations](https://mlsanity.com/paper/1412.5836/) (2014)
- [Designing an Effective Metric Learning Pipeline for Speaker Diarization](https://mlsanity.com/paper/1811.00183/) (2018)
- [Evaluation of taxonomic and neural embedding methods for calculating semantic similarity](https://mlsanity.com/paper/2209.15197/) (2022)
- [Integrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting](https://mlsanity.com/paper/1912.06889/) (2019)

## Built on by (most cited first, 1 of 1)

- [Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models](https://mlsanity.com/paper/2609.26610/) (2026)

## Abstract (arXiv)

Current breakthroughs in natural language processing have benefited dramatically from neural language models, through which distributional semantics can leverage neural data representations to facilitate downstream applications. Since neural embeddings use context prediction on word co-occurrences to yield dense vectors, they are inevitably prone to capture more semantic association than semantic similarity. To improve vector space models in deriving semantic similarity, we post-process neural word embeddings through deep metric learning, through which we can inject lexical-semantic relations, including syn/antonymy and hypo/hypernymy, into a distributional space. We introduce hierarchy-fitting, a novel semantic specialization approach to modelling semantic similarity nuances inherently stored in the IS-A hierarchies. Hierarchy-fitting attains state-of-the-art results on the common- and rare-word benchmark datasets for deriving semantic similarity from neural word embeddings. It also incorporates an asymmetric distance function to specialize hypernymy's directionality explicitly, through which it significantly improves vanilla embeddings in multiple evaluation tasks of detecting hypernymy and directionality without negative impacts on semantic similarity judgement. The results demonstrate the efficacy of hierarchy-fitting in specializing neural embeddings with semantic relations in late fusion, potentially expanding its applicability to aggregating heterogeneous data and various knowledge resources for learning multimodal semantic spaces.