mlsanityData to 7 Oct 2026

On the Opportunities and Risks of Foundation Models

arXiv 2108.07258 ↗First posted 16 Aug 2021cs.LG cs.AI cs.CYcited by 3,961

Not on the map. Why?

Cited by 175 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities,and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.

Key facts. Cited by 3,961 papers in the map (data through 7 Oct 2026). Builds on 1 of its 2 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

1 of 2 references.

References · 1

  1. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 400; 2023: 920; 2024: 1023; 2025: 927; 2026: 609.

3,961 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  13. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  14. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  17. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  18. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  19. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  20. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  21. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  22. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  23. 2022Red Teaming Language Models with Language Modelscited by 983
  24. 2023BloombergGPT: A Large Language Model for Financecited by 923
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  26. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  27. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  28. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  29. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  30. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  31. 2022A Generalist Agentcited by 802
  32. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  33. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  34. 2023StarCoder: may the source be with you!cited by 790
  35. 2021Robust fine-tuning of zero-shot modelscited by 785
  36. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  37. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  38. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733

50 most cited of 3,961 All 3,961 (JSON)