mlsanityData to 14 Oct 2026

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

arXiv 2002.08910 ↗First posted 10 Feb 2020cs.CL cs.LG stat.MLcited by 719

Not on the map. Why?

Cited by 52 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. To facilitate reproducibility and future work, we release our code and trained models at https://goo.gle/t5-cbqa.

Key facts. Cited by 719 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 116; 2022: 131; 2023: 148; 2024: 143; 2025: 97; 2026: 83.

719 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
38 more
  1. 2022Mass-Editing Memory in a Transformercited by 1,001
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  4. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  5. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  8. 2021Editing Factual Knowledge in Language Modelscited by 553
  9. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  11. 2023Active Retrieval Augmented Generationcited by 516
  12. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  13. 2021Fast Model Editing at Scalecited by 498
  14. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  15. 2022Memory-Based Model Editing at Scalecited by 447
  16. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  18. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  19. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  20. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  21. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  22. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  23. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  24. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  25. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  26. 2022Language Models are Realistic Tabular Data Generatorscited by 294
  27. 2022Teaching language models to support answers with verified quotescited by 292
  28. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  29. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  30. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  31. 2022Prompting GPT-3 To Be Reliablecited by 260
  32. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  33. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  34. 2022RealTime QA: What's the Answer Right Now?cited by 244
  35. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  36. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  37. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  38. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225

50 most cited of 719 All 719 (JSON)