mlsanityData to 7 Oct 2026

Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

arXiv 1905.01969 ↗First posted 22 Apr 2019cs.CL cs.AIcited by 164

Not on the map. Why?

Abstract (arXiv)

The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 45; 2023: 34; 2024: 20; 2025: 14; 2026: 11.

164 papers.

Sort:

By area:

  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  7. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  8. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  9. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  10. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  11. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  12. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
38 more
  1. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  2. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  3. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  4. 2023Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generationcited by 54
  5. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  6. 2022RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Groundingcited by 47
  7. 2021Open-book Video Captioning with Retrieve-Copy-Generate Networkcited by 43
  8. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  9. 2021Understanding Hard Negatives in Noise Contrastive Estimationcited by 39
  10. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  11. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  12. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  13. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  14. 2022Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterancescited by 32
  15. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  16. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  17. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  18. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
  19. 2022Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Modelscited by 26
  20. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  21. 2021A Proposed Conceptual Framework for a Representational Approach to Information Retrievalcited by 24
  22. 2022Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learningcited by 24
  23. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  24. 2021Entity Linking and Discovery via Arborescence-based Supervised Clusteringcited by 22
  25. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  26. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  27. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  28. 2022Improving Biomedical Information Retrieval with Neural Retrieverscited by 20
  29. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  30. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 20
  31. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  32. 2022Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervisioncited by 18
  33. 2021Hybrid Autoregressive Inference for Scalable Multi-hop Explanation Regenerationcited by 17
  34. 2022PMC-Patients: A Large-scale Dataset of Patient Summaries and Relations for Benchmarking Retrieval-based Clinical Decision Support Systemscited by 17
  35. 2023Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialoguescited by 17
  36. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  37. 2023Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understandingcited by 16
  38. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14

50 most cited of 164 All 164 (JSON)