mlsanityData to 7 Oct 2026

Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding

arXiv 2404.14600 ↗First posted 22 Apr 2024cs.IR cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

This paper introduces PAG-a novel optimization and decoding approach that guides autoregressive generation of document identifiers in generative retrieval models through simultaneous decoding. To this aim, PAG constructs a set-based and sequential identifier for each document. Motivated by the bag-of-words assumption in information retrieval, the set-based identifier is built on lexical tokens. The sequential identifier, on the other hand, is obtained via quantizing relevance-based representations of documents. Extensive experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin (e.g., 15.6% MRR improvements on MS MARCO), while achieving 22x speed up in terms of query latency.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 56 of its 90 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

56 of 90 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2019Passage Re-ranking with BERTcited by 629
38 more
  1. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  2. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  3. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  4. 2020Overview of the TREC 2019 deep learning trackcited by 282
  5. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  6. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021Overview of the TREC 2020 deep learning trackcited by 193
  9. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  10. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  11. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  12. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  13. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  14. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  15. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  16. 2022A Neural Corpus Indexer for Document Retrievalcited by 107
  17. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
  18. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  19. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  20. 2023Learning to Tokenize for Generative Retrievalcited by 66
  21. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  22. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  23. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  24. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  25. 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
  26. 2022Retrieval-Enhanced Machine Learningcited by 46
  27. 2023Learning to Rank in Generative Retrievalcited by 46
  28. 2022GERE: Generative Evidence Retrieval for Fact Verificationcited by 38
  29. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  30. 2023Scalable and Effective Generative Information Retrievalcited by 38
  31. 2018Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighborscited by 37
  32. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  33. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  34. 2022Ultron: An Ultimate Retriever on Corpus with a Model-based Indexercited by 30
  35. 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
  36. 2023Continual Learning for Generative Retrieval over Dynamic Corporacited by 25
  37. 2023TOME: A Two-stage Approach for Model-based Retrievalcited by 21
  38. 2023Language Models As Semantic Indexerscited by 20

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  2. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  3. 2025Lightweight and Direct Document Relevance Optimization for Generative Information Retrievalcited by 9
  4. 2025LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generationcited by 5
  5. 2025Exploring Training and Inference Scaling Laws in Generative Retrievalcited by 4
  6. 2025SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMscited by 4
  7. 2025MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corporacited by 4
  8. 2024Generative Retrieval with Few-shot Indexingcited by 3
  9. 2025DiffuGR: Generative Document Retrieval with Diffusion Language Modelscited by 2
  10. 2025Foundations of GenIRcited by 1
  11. 2025Retrieval-in-the-Chain: Bootstrapping Large Language Models for Generative Retrievalcited by 1
  12. 2026BEAR: Towards Beam-Search-Aware Optimization for Recommendation with Large Language Modelscited by 1
7 more
  1. 2026From Relevance to Authority: Authority-aware Generative Retrieval in Web Search Enginescited by 1
  2. 2026A Parametric Memory Head for Continual Generative Retrievalcited by 1
  3. 2026It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learningcited by 1
  4. 2026Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrievalcited by 0
  5. 2026UniGD: A Unified Generative-Discriminative Framework for Industrial Retrievalcited by 0
  6. 2026Generative Universal Multimodal Retrieval with Dual-role Identifierscited by 0
  7. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0