# On the Societal Impact of Open Foundation Models

arXiv: https://arxiv.org/abs/2403.07918 · HTML: https://mlsanity.com/paper/2403.07918/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 138 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 30 of 48)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://mlsanity.com/paper/1802.07228/) (2018)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021)
- [Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://mlsanity.com/paper/2210.17546/) (2022)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Discrimination in Online Ad Delivery](https://mlsanity.com/paper/1301.6822/) (2013)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)

## Built on by (most cited first, 30 of 65)

- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions](https://mlsanity.com/paper/2411.05025/) (2024)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [The Jailbreak Tax: How Useful are Your Jailbreak Outputs?](https://mlsanity.com/paper/2504.10694/) (2025)
- [COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act](https://mlsanity.com/paper/2410.07959/) (2024)
- [The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence](https://mlsanity.com/paper/2403.13784/) (2024)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [Aloe: A Family of Fine-tuned Open Healthcare LLMs](https://mlsanity.com/paper/2405.01886/) (2024)
- [The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub](https://mlsanity.com/paper/2405.13058/) (2024)
- [Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses](https://mlsanity.com/paper/2407.02551/) (2024)
- [HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router](https://mlsanity.com/paper/2410.02684/) (2024)
- [Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research](https://mlsanity.com/paper/2412.06966/) (2024)
- [Training Compute Thresholds: Features and Functions in AI Regulation](https://mlsanity.com/paper/2405.10799/) (2024)
- [Acceptable Use Policies for Foundation Models](https://mlsanity.com/paper/2409.09041/) (2024)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [Responsible Reporting for Frontier AI Development](https://mlsanity.com/paper/2404.02675/) (2024)
- [Societal Adaptation to Advanced AI](https://mlsanity.com/paper/2405.10295/) (2024)
- [Prioritizing High-Consequence Biological Capabilities in Evaluations of Artificial Intelligence Models](https://mlsanity.com/paper/2407.13059/) (2024)
- [Independence Tests for Language Models](https://mlsanity.com/paper/2502.12292/) (2025)
- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal](https://mlsanity.com/paper/2403.13309/) (2024)
- [The Reality of AI and Biorisk](https://mlsanity.com/paper/2412.01946/) (2024)
- [Build Agent Advocates, Not Platform Agents](https://mlsanity.com/paper/2505.04345/) (2025)
- [Dynamic Risk Assessments for Offensive Cybersecurity Agents](https://mlsanity.com/paper/2505.18384/) (2025)
- [Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence](https://mlsanity.com/paper/2405.15802/) (2024)
- [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025)
- [Sociotechnical Implications of Generative Artificial Intelligence for Information Access](https://mlsanity.com/paper/2405.11612/) (2024)

## Abstract (arXiv)

Foundation models are powerful technologies: how they are released publicly directly shapes their societal impact. In this position paper, we focus on open foundation models, defined here as those with broadly available model weights (e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties (e.g. greater customizability, poor monitoring) of open foundation models that lead to both their benefits and risks. Open foundation models present significant benefits, with some caveats, that span innovation, competition, the distribution of decision-making power, and transparency. To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons), we find that current research is insufficient to effectively characterize the marginal risk of open foundation models relative to pre-existing technologies. The framework helps explain why the marginal risk is low in some cases, clarifies disagreements about misuse risks by revealing that past work has focused on different subsets of the framework with different assumptions, and articulates a way forward for more constructive debate. Overall, our work helps support a more grounded assessment of the societal impact of open foundation models by outlining what research is needed to empirically validate their theoretical benefits and risks.