# Relevance-based Word Embedding

arXiv: https://arxiv.org/abs/1705.03556 · HTML: https://mlsanity.com/paper/1705.03556/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2017.

## Built on by (most cited first, 12 of 12)

- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Retrieval-Enhanced Machine Learning](https://mlsanity.com/paper/2205.01230/) (2022)
- [Context Aware Query Rewriting for Text Rankers using LLM](https://mlsanity.com/paper/2308.16753/) (2023)
- [Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2306.16478/) (2023)
- [ESCORT: Ethereum Smart COntRacTs Vulnerability Detection using Deep Neural Network and Transfer Learning](https://mlsanity.com/paper/2103.12607/) (2021)
- [Test-time Corpus Feedback: From Retrieval to RAG](https://mlsanity.com/paper/2508.15437/) (2025)
- [Event-Centric Query Expansion in Web Search](https://mlsanity.com/paper/2305.19019/) (2023)
- [Emotion Detection in Twitter Messages Using Combination of Long Short-Term Memory and Convolutional Deep Neural Networks](https://mlsanity.com/paper/2503.20163/) (2025)
- [Search Intenion Network for Personalized Query Auto-Completion in E-Commerce](https://mlsanity.com/paper/2403.02609/) (2024)
- [Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model](https://mlsanity.com/paper/2307.00370/) (2023)
- [The Surprising Effectiveness of Rankers Trained on Expanded Queries](https://mlsanity.com/paper/2404.02587/) (2024)
- [Forgotten History or Test-of-Time? Retrospect and Prospect on RAG from an IR Perspective](https://mlsanity.com/paper/2608.08445/) (2026)

## Abstract (arXiv)

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.