mlsanityData to 14 Oct 2026

Risks and Opportunities of Open-Source Generative AI

arXiv 2405.08597 ↗First posted 14 May 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Applications of Generative AI (Gen AI) are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about the potential risks of the technology, and resulted in calls for tighter regulation, in particular from some of the major tech companies who are leading in AI development. This regulation is likely to put at risk the budding field of open-source generative AI. Using a three-stage framework for Gen AI development (near, mid and long-term), we analyze the risks and opportunities of open-source generative AI models with similar capabilities to the ones currently available (near to mid-term) and with greater capabilities (long-term). We argue that, overall, the benefits of open-source Gen AI outweigh its risks. As such, we encourage the open sourcing of models, training and evaluation data, and provide a set of recommendations and best practices for managing risks associated with open-source generative AI.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 91 of its 203 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

91 of 203 references.

References · 91

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  13. 2019Green AIcited by 627
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  15. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  16. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  17. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  18. 2020Artificial Intelligence, Values and Alignmentcited by 448
  19. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  20. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  21. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  22. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  23. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  24. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  25. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  26. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  27. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  28. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  29. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  30. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  31. 2023An Overview of Catastrophic AI Riskscited by 222
  32. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  33. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  34. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  35. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  36. 2023Model evaluation for extreme riskscited by 177
  37. 2021Alignment of Language Agentscited by 168
  38. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Large Language Model Safety: A Holistic Surveycited by 59
  2. 2024Can Editing LLMs Inject Harm?cited by 22
  3. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 20
  4. 2024The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hubcited by 18
  5. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 14
  6. 2024Acceptable Use Policies for Foundation Modelscited by 12
  7. 2025Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoringcited by 7
  8. 2025MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AIcited by 7
  9. 2025Red Teaming AI Policy: A Taxonomy of Avoision and the EU AI Actcited by 6
  10. 2024Comparative Global AI Regulation: Policy Perspectives from the EU, China, and the UScited by 5
  11. 2025AI Governance through Marketscited by 5
  12. 2025Opening the Scope of Openness in AIcited by 0
1 more
  1. 2026Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitationscited by 0