# Participation in the age of foundation models

arXiv: https://arxiv.org/abs/2405.19479 · HTML: https://mlsanity.com/paper/2405.19479/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 106 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice](https://mlsanity.com/paper/2310.00907/) (2023)
- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain](https://mlsanity.com/paper/2309.08133/) (2023)
- [Bringing the People Back In: Contesting Benchmark Machine Learning Datasets](https://mlsanity.com/paper/2007.07399/) (2020)
- [Queer In AI: A Case Study in Community-Led Participatory AI](https://mlsanity.com/paper/2303.16972/) (2023)
- [Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision](https://mlsanity.com/paper/2007.14886/) (2020)
- [Designing Participatory AI: Creative Professionals' Worries and Expectations about Generative AI](https://mlsanity.com/paper/2303.08931/) (2023)
- [Democratic Policy Development using Collective Dialogues and AI](https://mlsanity.com/paper/2311.02242/) (2023)
- [The Data-Production Dispositif](https://mlsanity.com/paper/2205.11963/) (2022)
- [Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks](https://mlsanity.com/paper/2011.02511/) (2020)
- [Going public: the role of public participation approaches in commercial AI labs](https://mlsanity.com/paper/2306.09871/) (2023)
- [Using large language models to promote health equity](https://mlsanity.com/paper/2312.14804/) (2023)

## Built on by (most cited first, 30 of 32)

- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Acceptable Use Policies for Foundation Models](https://mlsanity.com/paper/2409.09041/) (2024)
- [HumT DumT: Measuring and controlling human-like language in LLMs](https://mlsanity.com/paper/2502.13259/) (2025)
- ["Ownership, Not Just Happy Talk": Co-Designing a Participatory Large Language Model for Journalism](https://mlsanity.com/paper/2501.17299/) (2025)
- [AI Supply Chains: An Emerging Ecosystem of AI Actors, Products, and Services](https://mlsanity.com/paper/2504.20185/) (2025)
- [Just-In-Time Objectives: A General Approach for Specialized AI Interactions](https://mlsanity.com/paper/2510.14591/) (2025)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [Decentralising LLM Alignment: A Case for Context, Pluralism, and Participation](https://mlsanity.com/paper/2509.08858/) (2025)
- [Attention to Non-Adopters](https://mlsanity.com/paper/2510.15951/) (2025)
- [Data Caricatures: On the Representation of African American Language in Pretraining Corpora](https://mlsanity.com/paper/2503.10789/) (2025)
- [Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency](https://mlsanity.com/paper/2506.07281/) (2025)
- [Bringing AI Participation Down to Scale: A Comment on Open AIs Democratic Inputs to AI Project](https://mlsanity.com/paper/2407.11613/) (2024)
- [Nteasee: Understanding Needs in AI for Health in Africa -- A Mixed-Methods Study of Expert and General Population Perspectives](https://mlsanity.com/paper/2409.12197/) (2024)
- [ModelCitizens: Representing Community Voices in Online Safety](https://mlsanity.com/paper/2507.05455/) (2025)
- [ONION: A Multi-Layered Framework for Participatory ER Design](https://mlsanity.com/paper/2507.08702/) (2025)
- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)
- [Towards Meaningful Transparency in Civic AI Systems](https://mlsanity.com/paper/2510.07889/) (2025)
- [Interrogating Design Homogenization in Web Vibe Coding](https://mlsanity.com/paper/2603.13036/) (2026)
- [Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South](https://mlsanity.com/paper/2605.19190/) (2026)
- [AI red-teaming is a sociotechnical problem: on values, labor, and harms](https://mlsanity.com/paper/2412.09751/) (2024)
- [How do datasets, developers, and models affect biases in a low-resourced language?: The Case of the Bengali Language](https://mlsanity.com/paper/2506.06816/) (2025)
- [PolicyPad: Collaborative Prototyping of LLM Policies](https://mlsanity.com/paper/2509.19680/) (2025)
- [Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners](https://mlsanity.com/paper/2511.05501/) (2025)
- [Cripping AI: Reimagining AI Through Lived Disability Experiences](https://mlsanity.com/paper/2605.02080/) (2026)
- [AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligence](https://mlsanity.com/paper/2605.16291/) (2026)
- [AI From the Margins (AIM): Rethinking Participatory AI Design Through the Lived Experience of Minoritized Communities](https://mlsanity.com/paper/2606.01171/) (2026)
- [Framing Migration News with LLMs: Structured CoT as a Support for Human Interpretation](https://mlsanity.com/paper/2606.03761/) (2026)
- [Can Data Work be Reparative?](https://mlsanity.com/paper/2606.09408/) (2026)

## Abstract (arXiv)

Growing interest and investment in the capabilities of foundation models has positioned such systems to impact a wide array of public services. Alongside these opportunities is the risk that these systems reify existing power imbalances and cause disproportionate harm to marginalized communities. Participatory approaches hold promise to instead lend agency and decision-making power to marginalized stakeholders. But existing approaches in participatory AI/ML are typically deeply grounded in context - how do we apply these approaches to foundation models, which are, by design, disconnected from context? Our paper interrogates this question. First, we examine existing attempts at incorporating participation into foundation models. We highlight the tension between participation and scale, demonstrating that it is intractable for impacted communities to meaningfully shape a foundation model that is intended to be universally applicable. In response, we develop a blueprint for participatory foundation models that identifies more local, application-oriented opportunities for meaningful participation. In addition to the "foundation" layer, our framework proposes the "subfloor'' layer, in which stakeholders develop shared technical infrastructure, norms and governance for a grounded domain, and the "surface'' layer, in which affected communities shape the use of a foundation model for a specific downstream task. The intermediate "subfloor'' layer scopes the range of potential harms to consider, and affords communities more concrete avenues for deliberation and intervention. At the same time, it avoids duplicative effort by scaling input across relevant use cases. Through three case studies in clinical care, financial services, and journalism, we illustrate how this multi-layer model can create more meaningful opportunities for participation than solely intervening at the foundation layer.