mlsanityData to 7 Oct 2026

In Defense of Cross-Encoders for Zero-Shot Retrieval

arXiv 2212.06121 ↗First posted 12 Dec 2022cs.IR cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Bi-encoders and cross-encoders are widely used in many state-of-the-art retrieval pipelines. In this work we study the generalization ability of these two types of architectures on a wide range of parameter count on both in-domain and out-of-domain scenarios. We find that the number of parameters and early query-document interactions of cross-encoders play a significant role in the generalization ability of retrieval models. Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning. Furthermore, we show that cross-encoders largely outperform bi-encoders of similar size in several tasks. In the BEIR benchmark, our largest cross-encoder surpasses a state-of-the-art bi-encoder by more than 4 average points. Finally, we show that using bi-encoders as first-stage retrievers provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks. The code is available at https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

41 of 57 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2019Passage Re-ranking with BERTcited by 629
  10. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
29 more
  1. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  2. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  3. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  4. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  5. 2019Document Expansion by Query Predictioncited by 225
  6. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021Overview of the TREC 2020 deep learning trackcited by 193
  9. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  10. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  11. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  12. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  13. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  14. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  15. 2021Learning Passage Impacts for Inverted Indexescited by 68
  16. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  17. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  18. 2021Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationscited by 60
  19. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  20. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  21. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  22. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  23. 2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
  24. 2022A Thorough Examination on Zero-shot Dense Retrievalcited by 21
  25. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  26. 2020Flexible retrieval with NMSLIB and FlexNeuARTcited by 14
  27. 2020Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learnedcited by 10
  28. 2022Scalable Training of Language Models using JAX pjit and TPUv4cited by 9
  29. 2022On Survivorship Bias in MS MARCOcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 7; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasetscited by 25
  2. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMscited by 18
  3. 2023InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankerscited by 14
  4. 2024D2LLM: Decomposed and Distilled Large Language Models for Semantic Searchcited by 9
  5. 2025GLiClass: Generalist Lightweight Model for Sequence Classification Taskscited by 9
  6. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  7. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelinescited by 7
  8. 2024Assessing generalization capability of text ranking models in Polishcited by 3
  9. 2023ExaRanker: Explanation-Augmented Neural Rankercited by 2
  10. 2023LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Biascited by 2
  11. 2025Beyond Retrieval: Ensembling Cross-Encoders and GPT Rerankers with LLMs for Biomedical QAcited by 2
  12. 2025Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMscited by 2
9 more
  1. 2023Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoderscited by 1
  2. 2024Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoderscited by 1
  3. 2025Towards a scalable AI-driven framework for data-independent Cyber Threat Intelligence Information Extractioncited by 1
  4. 2025Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Databasecited by 1
  5. 2026Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generationcited by 1
  6. 2023A General-Purpose Multilingual Document Encodercited by 0
  7. 2025Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Miningcited by 0
  8. 2026Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Rerankingcited by 0
  9. 2026SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scalecited by 0