# Beyond [CLS] through Ranking by Generation

arXiv: https://arxiv.org/abs/2010.03073 · HTML: https://mlsanity.com/paper/2010.03073/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [NaturalProofs: Mathematical Theorem Proving in Natural Language](https://mlsanity.com/paper/2104.01112/) (2021)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [Questions Are All You Need to Train a Dense Passage Retriever](https://mlsanity.com/paper/2206.10658/) (2022)
- [A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems](https://mlsanity.com/paper/2204.06923/) (2022)
- [Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker](https://mlsanity.com/paper/2305.13729/) (2023)
- [Few-shot Reranking for Multi-hop QA via Language Model Prompting](https://mlsanity.com/paper/2205.12650/) (2022)
- [Generative Prompt Tuning for Relation Classification](https://mlsanity.com/paper/2210.12435/) (2022)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)
- [ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference](https://mlsanity.com/paper/2204.11458/) (2022)
- [Learning List-Level Domain-Invariant Representations for Ranking](https://mlsanity.com/paper/2212.10764/) (2022)
- [Generative Context Pair Selection for Multi-hop Question Answering](https://mlsanity.com/paper/2104.08744/) (2021)
- [NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders](https://mlsanity.com/paper/2305.14499/) (2023)
- [Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation](https://mlsanity.com/paper/2309.16599/) (2023)
- [Improving Generated and Retrieved Knowledge Combination Through Zero-shot Generation](https://mlsanity.com/paper/2412.18800/) (2024)

## Abstract (arXiv)

Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document's language model, were very successful in various IR tasks in the past. However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead. Recently, deep generative models such as GPT2 and BART have been shown to be excellent text generators, but their effectiveness as rankers have not been demonstrated yet. In this work, we revisit the generative framework for information retrieval and show that our generative approaches are as effective as state-of-the-art semantic similarity-based discriminative models for the answer selection task. Additionally, we demonstrate the effectiveness of unlikelihood losses for IR.