mlsanityData to 14 Oct 2026

Document Ranking with a Pretrained Sequence-to-Sequence Model

arXiv 2003.06713 ↗First posted 14 Mar 2020cs.IR cs.LGcited by 376

Not on the map. Why?

Cited by 82 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work proposes a novel adaptation of a pretrained sequence-to-sequence model to the task of document ranking. Our approach is fundamentally different from a commonly-adopted classification-based formulation of ranking, based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words", and how the underlying logits of these target words can be interpreted as relevance probabilities for ranking. On the popular MS MARCO passage ranking task, experimental results show that our approach is at least on par with previous classification-based models and can surpass them with larger, more-recent models. On the test collection from the TREC 2004 Robust Track, we demonstrate a zero-shot transfer-based approach that outperforms previous state-of-the-art models requiring in-dataset cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only model in a data-poor regime (i.e., with few training examples). We investigate this observation further by varying target words to probe the model's use of latent knowledge.

Key facts. Cited by 376 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 30; 2022: 33; 2023: 54; 2024: 87; 2025: 76; 2026: 96.

376 papers.

Sort:

By area:

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  3. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  4. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  8. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  9. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 165
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  12. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
38 more
  1. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  2. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  3. 2022Task-aware Retrieval with Instructionscited by 107
  4. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  5. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  6. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 83
  7. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  8. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 82
  9. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  10. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  13. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 70
  14. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  15. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  16. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 62
  17. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 60
  18. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  19. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 55
  20. 2023Scientific Fact-Checking: A Survey of Resources and Approachescited by 54
  21. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  22. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  23. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 48
  24. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  25. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  26. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  27. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
  28. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  29. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  30. 2021Joint Passage Ranking for Diverse Multi-Answer Retrievalcited by 43
  31. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 43
  32. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 40
  33. 2022GERE: Generative Evidence Retrieval for Fact Verificationcited by 38
  34. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  35. 2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
  36. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  37. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  38. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generatorcited by 34

50 most cited of 376 All 376 (JSON)