# Multivariate Representation Learning for Information Retrieval

arXiv: https://arxiv.org/abs/2304.14522 · HTML: https://mlsanity.com/paper/2304.14522/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 69 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2010.02666/) (2020)
- [Order-Embeddings of Images and Language](https://mlsanity.com/paper/1511.06361/) (2015)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval](https://mlsanity.com/paper/1910.10687/) (2019)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [Distilling Dense Representations for Ranking using Tightly-Coupled Teachers](https://mlsanity.com/paper/2010.11386/) (2020)
- [Retrieval-Enhanced Machine Learning](https://mlsanity.com/paper/2205.01230/) (2022)
- [Curriculum Learning for Dense Retrieval Distillation](https://mlsanity.com/paper/2204.13679/) (2022)
- [On the Calibration and Uncertainty of Neural Learning to Rank Models](https://mlsanity.com/paper/2101.04356/) (2021)

## Built on by (most cited first, 3 of 3)

- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Bridging Personalization and Control in Scientific Personalized Search](https://mlsanity.com/paper/2411.02790/) (2024)
- [Ontology-driven personalized information retrieval for XML documents](https://mlsanity.com/paper/2603.21139/) (2026)

## Abstract (arXiv)

Dense retrieval models use bi-encoder network architectures for learning query and document representations. These representations are often in the form of a vector representation and their similarities are often computed using the dot product function. In this paper, we propose a new representation learning framework for dense retrieval. Instead of learning a vector for each query and document, our framework learns a multivariate distribution and uses negative multivariate KL divergence to compute the similarity between distributions. For simplicity and efficiency reasons, we assume that the distributions are multivariate normals and then train large language models to produce mean and variance vectors for these distributions. We provide a theoretical foundation for the proposed framework and show that it can be seamlessly integrated into the existing approximate nearest neighbor algorithms to perform retrieval efficiently. We conduct an extensive suite of experiments on a wide range of datasets, and demonstrate significant improvements compared to competitive dense retrieval models.