mlsanityData to 7 Oct 2026

What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary

arXiv 2212.10380 ↗First posted 20 Dec 2022cs.CL cs.IRcited by 28

Not on the map. Why?

Abstract (arXiv)

Dual encoders are now the dominant architecture for dense retrieval. Yet, we have little understanding of how they represent text, and why this leads to good performance. In this work, we shed light on this question via distributions over the vocabulary. We propose to interpret the vector representations produced by dual encoders by projecting them into the model's vocabulary space. We show that the resulting projections contain rich semantic information, and draw connection between them and sparse retrieval. We find that this view can offer an explanation for some of the failure cases of dense retrievers. For example, we observe that the inability of models to handle tail entities is correlated with a tendency of the token distributions to forget some of the tokens of those entities. We leverage this insight and propose a simple way to enrich query and passage representations with lexical information at inference time, and show that this significantly improves performance compared to the original model in zero-shot settings, and specifically on the BEIR benchmark.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

46 of 71 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
34 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  9. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  10. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  11. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  12. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  13. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  14. 2019Document Expansion by Query Predictioncited by 225
  15. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  16. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  17. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  18. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  19. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  20. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  21. 2022Analyzing Transformers in Embedding Spacecited by 131
  22. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  23. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  24. 2020Learning Dense Representations of Phrases at Scalecited by 91
  25. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  26. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  27. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  28. 2021Learning to Retrieve Passages without Supervisioncited by 46
  29. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  30. 2020Expansion via Prediction of Importance with Contextualizationcited by 26
  31. 2020ABNIRML: Analyzing the Behavior of Neural IR Modelscited by 23
  32. 2021Towards Robust Neural Retrieval Models with Synthetic Pre-Trainingcited by 9
  33. 2022Isotropic Representation Can Improve Dense Retrievalcited by 8
  34. 2022Decoding a Neural Retriever's Latent Space for Query Suggestioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 11; 2025: 7; 2026: 2.

28 papers.

Sort:

By area:

  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  2. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  3. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  4. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  5. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  6. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  7. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  8. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  9. 2023The Hidden Language of Diffusion Modelscited by 34
  10. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  11. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 30
  12. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
16 more
  1. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  2. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  3. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  4. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12
  5. 2024When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?cited by 11
  6. 2023RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Modelscited by 4
  7. 2024Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrievalcited by 3
  8. 2025EmoRAG: Evaluating RAG Robustness to Symbolic Perturbationscited by 3
  9. 2025Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewritingcited by 2
  10. 2025Event Causality Identification with Synthetic Controlcited by 2
  11. 2025Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layerscited by 2
  12. 2025ImpliRet: Benchmarking the Implicit Fact Retrieval Challengecited by 1
  13. 2024Semi-Parametric Retrieval via Binary Bag-of-Tokens Indexcited by 0
  14. 2025Comparing Lexical and Semantic Vector Search Methods When Classifying Medical Documentscited by 0
  15. 2026Masking or Mitigating? Deconstructing the Impact of Query Rewriting on Retriever Biases in RAGcited by 0
  16. 2026E-SENS: Exclusion-Sensitive Penalization for Negative-Constraint Retrievalcited by 0