mlsanityData to 7 Oct 2026

Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

arXiv 2212.02027 ↗First posted 5 Dec 2022cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Systems for knowledge-intensive tasks such as open-domain question answering (QA) usually consist of two stages: efficient retrieval of relevant documents from a large corpus and detailed reading of the selected documents to generate answers. Retrievers and readers are usually modeled separately, which necessitates a cumbersome implementation and is hard to train and adapt in an end-to-end fashion. In this paper, we revisit this design and eschew the separate architecture and training in favor of a single Transformer that performs Retrieval as Attention (ReAtt), and end-to-end training solely based on supervision from the end QA task. We demonstrate for the first time that a single model trained end-to-end can achieve both competitive retrieval and QA performance, matching or slightly outperforming state-of-the-art separately trained retrievers and readers. Moreover, end-to-end adaptation significantly boosts its performance on out-of-domain datasets in both supervised and unsupervised settings, making our model a simple and adaptable solution for knowledge-intensive tasks. Code and models are available at https://github.com/jzbjyb/ReAtt.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
27 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  6. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  7. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  8. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  9. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  10. 2022Memorizing Transformerscited by 219
  11. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  12. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  13. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  14. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  15. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  16. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  17. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  18. 2022Training Language Models with Memory Augmentationcited by 112
  19. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  20. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  21. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  22. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  23. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  24. 2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
  25. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  26. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  27. 2021You Only Need One Model for Open-domain Question Answeringcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 18; 2025: 10; 2026: 2.

45 papers.

Sort:

By area:

  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023Active Retrieval Augmented Generationcited by 506
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  4. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  5. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  6. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  7. 2023Retrieval meets Long Context Large Language Modelscited by 114
  8. 2024Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Modelscited by 84
  9. 2024DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Modelscited by 63
  10. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  11. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  12. 2024PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makerscited by 42
33 more
  1. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  2. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  3. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  4. 2024Metacognitive Retrieval-Augmented Large Language Modelscited by 26
  5. 2024Manipulating Large Language Models to Increase Product Visibilitycited by 24
  6. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 24
  7. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  8. 2024An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaborationcited by 16
  9. 2025SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generationcited by 14
  10. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  11. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  12. 2023Robust Natural Language Understanding with Residual Attention Debiasingcited by 10
  13. 2024RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generationcited by 10
  14. 2023GLIMMER: generalized late-interaction memory rerankercited by 7
  15. 20231-PAGER: One Pass Answer Generation and Evidence Retrievalcited by 7
  16. 2025DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generationcited by 7
  17. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  18. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5
  19. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compressioncited by 4
  20. 2025IterKey: Iterative Keyword Generation with LLMs for Enhanced Retrieval Augmented Generationcited by 4
  21. 2024Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervisioncited by 3
  22. 2025Dynamic and Parametric Retrieval-Augmented Generationcited by 3
  23. 2025MultiRAG: A Knowledge-guided Framework for Mitigating Hallucination in Multi-source Retrieval Augmented Generationcited by 3
  24. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformercited by 2
  25. 2024Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generationcited by 2
  26. 2024FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selectioncited by 2
  27. 2023Pay Attention to What You Needcited by 1
  28. 2024You Only Use Reactive Attention Slice For Long Context Retrievalcited by 1
  29. 2025Equipping Retrieval-Augmented Large Language Models with Document Structure Awarenesscited by 1
  30. 2026ACR: Adaptive Context Refactoring via Context Refactoring Operators for Multi-Turn Dialoguecited by 1
  31. 2024Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorizationcited by 0
  32. 2025Few-Shot Multilingual Open-Domain QA from 5 Examplescited by 0
  33. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0