mlsanityData to 7 Oct 2026

Atlas: Few-shot Learning with Retrieval Augmented Language Models

arXiv 2208.03299 ↗First posted 5 Aug 2022cs.CLcited by 940

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.

Key facts. Cited by 940 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 74 of its 94 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

74 of 94 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2016Matching Networks for One Shot Learningcited by 2,246
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  13. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  14. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  15. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  16. 2019How Can We Know What Language Models Know?cited by 953
  17. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  18. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  19. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  20. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  21. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  22. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  23. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  24. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  25. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  26. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  27. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  28. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  29. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  30. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  31. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  32. 2020Autoregressive Entity Retrievalcited by 333
  33. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  34. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  35. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  36. 2021How Many Data Points is a Prompt Worth?cited by 197
  37. 2020How Context Affects Language Models' Factual Predictionscited by 181
  38. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 185; 2024: 295; 2025: 243; 2026: 183.

940 papers.

Sort:

By area:

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
38 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  7. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  8. 2023Active Retrieval Augmented Generationcited by 506
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  10. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  11. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  13. 2023Augmented Language Models: a Surveycited by 398
  14. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  15. 2024The Faiss librarycited by 370
  16. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  17. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  18. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  19. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  20. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  21. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  22. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  23. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  24. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  25. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  26. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  27. 2023Tool Learning with Foundation Modelscited by 302
  28. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  29. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  30. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  31. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  32. 2022Prompting GPT-3 To Be Reliablecited by 258
  33. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  34. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  35. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  36. 2023Challenges and Applications of Large Language Modelscited by 251
  37. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  38. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243

50 most cited of 940 All 940 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)
  2. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questions
  3. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  4. 2025Coling-UniA at SciVQA 2025: Few-Shot Example Retrieval and Confidence-Informed Ensembling for Multimodal Large Language Models
  5. 2025AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
  6. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  7. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning
  8. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval