mlsanityData to 14 Oct 2026

RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models

arXiv 2308.07922 ↗First posted 15 Aug 2023cs.CL cs.AI cs.LGcited by 37

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

In this paper, we investigate the in-context learning ability of retrieval-augmented encoder-decoder language models. We first conduct a comprehensive analysis of existing models and identify their limitations in in-context learning, primarily due to a mismatch between pretraining and inference, as well as a restricted context length. To address these issues, we propose RAVEN, a model that combines retrieval-augmented masked language modeling and prefix language modeling. We further introduce Fusion-in-Context Learning to enhance the few-shot performance by enabling the model to leverage more in-context examples without requiring additional training. Through extensive experiments, we demonstrate that our simple yet effective design significantly improves performance, achieving results comparable to the most advanced language models in certain scenarios, despite having substantially fewer parameters. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning and encourages further research in this direction.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 42 of its 52 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

42 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  3. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP28 Dec
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  5. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inference27 May
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
24 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  12. 2019ELI5: Long Form Question Answeringcited by 575
  13. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  15. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  16. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  17. 2022UL2: Unifying Language Learning Paradigmscited by 295
  18. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  19. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  20. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  21. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  22. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  23. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  24. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 23; 2025: 8; 2026: 2.

37 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  3. 2023Retrieval meets Long Context Large Language Modelscited by 115
  4. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  5. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 71
  6. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  7. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 53
  8. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  9. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMscited by 24
  10. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 24
  11. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  12. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23
25 more
  1. 2024Natural Language Processing in the Patent Domain: A Surveycited by 20
  2. 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
  3. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  4. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  5. 2024Towards LifeSpan Cognitive Systemscited by 13
  6. 2024Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Studycited by 11
  7. 2024PlanGPT: Enhancing Urban Planning with Tailored Language Model and Efficient Retrievalcited by 8
  8. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  9. 2024Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learningcited by 6
  10. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  11. 2024TorchOpera: A Compound AI System for LLM Safetycited by 4
  12. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compressioncited by 4
  13. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  14. 2024The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMscited by 3
  15. 2025RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentationcited by 3
  16. 2025ARC-Encoder: learning compressed text representations for large language modelscited by 3
  17. 2024WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenariocited by 2
  18. 2024FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inferencecited by 1
  19. 2025DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generationcited by 1
  20. 2024Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorizationcited by 0
  21. 2024Retrieval-Augmented Language Model for Extreme Multi-Label Knowledge Graph Link Predictioncited by 0
  22. 2025RE2: Improving Chinese Grammatical Error Correction via Retrieving Appropriate Examples with Explanationcited by 0
  23. 2025Chain of Summaries: Summarization Through Iterative Questioningcited by 0
  24. 2026CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMscited by 0
  25. 2026Retrieval-Based In-Context Learning: A Domain Adaptation Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RARe: Retrieval Augmented Retrieval with In-Context Examples
  2. 2025Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture
  3. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  4. 2022Decoupled Context Processing for Context Augmented Language Modeling
  5. 2023Learning to Retrieve In-Context Examples for Large Language Models
  6. 2024Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models
  7. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  8. 2023In-context Autoencoder for Context Compression in a Large Language Model