# Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governance

arXiv: https://arxiv.org/abs/2408.01458 · HTML: https://mlsanity.com/paper/2408.01458/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 209 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022)
- [How Different Groups Prioritize Ethical Values for Responsible AI](https://mlsanity.com/paper/2205.07722/) (2022)
- [Supporting Human-AI Collaboration in Auditing LLMs with LLMs](https://mlsanity.com/paper/2304.09991/) (2023)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [Queer In AI: A Case Study in Community-Led Participatory AI](https://mlsanity.com/paper/2303.16972/) (2023)
- [Envisioning Communities: A Participatory Approach Towards AI for Social Good](https://mlsanity.com/paper/2105.01774/) (2021)
- [WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?](https://mlsanity.com/paper/2305.06415/) (2023)
- [Whose AI Dream? In search of the aspiration in data annotation](https://mlsanity.com/paper/2203.10748/) (2022)
- [Long-Term Trends in the Public Perception of Artificial Intelligence](https://mlsanity.com/paper/1609.04904/) (2016)
- [Exciting, Useful, Worrying, Futuristic: Public Perception of Artificial Intelligence in 8 Countries](https://mlsanity.com/paper/2001.00081/) (2019)
- [Representativeness in Statistics, Politics, and Machine Learning](https://mlsanity.com/paper/2101.03827/) (2021)
- [A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Research](https://mlsanity.com/paper/2207.04171/) (2022)
- [U.S. Public Opinion on the Governance of Artificial Intelligence](https://mlsanity.com/paper/1912.12835/) (2019)
- [Going public: the role of public participation approaches in commercial AI labs](https://mlsanity.com/paper/2306.09871/) (2023)
- [On Microtargeting Socially Divisive Ads: A Case Study of Russia-Linked Ad Campaigns on Facebook](https://mlsanity.com/paper/1808.09218/) (2018)
- [Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics](https://mlsanity.com/paper/1812.05948/) (2018)
- [Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms](https://mlsanity.com/paper/2307.10223/) (2023)
- [Situated Data, Situated Systems: A Methodology to Engage with Power Relations in Natural Language Processing Research](https://mlsanity.com/paper/2011.05911/) (2020)
- [Risks of AI Foundation Models in Education](https://mlsanity.com/paper/2110.10024/) (2021)
- [A Systematic Literature Review of Human-Centered, Ethical, and Responsible AI](https://mlsanity.com/paper/2302.05284/) (2023)
- [Human participants in AI research: Ethics and transparency in practice](https://mlsanity.com/paper/2311.01254/) (2023)
- [Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study](https://mlsanity.com/paper/2305.18307/) (2023)
- [Implications of Regulations on the Use of AI and Generative AI for Human-Centered Responsible Artificial Intelligence](https://mlsanity.com/paper/2403.00148/) (2024)

## Built on by (most cited first, 4 of 4)

- [Understanding Gender Bias in AI-Generated Product Descriptions](https://mlsanity.com/paper/2506.05390/) (2025)
- [The Hall of AI Fears and Hopes: Comparing the Views of AI Influencers and those of Members of the U.S. Public Through an Interactive Platform](https://mlsanity.com/paper/2504.06016/) (2025)
- [An Exploratory Study on LLM-Generated Code and Comments in Code Repositories](https://mlsanity.com/paper/2607.01867/) (2026)
- [Designing the Future of User Feedback for Generative AI](https://mlsanity.com/paper/2610.02631/) (2026)

## Abstract (arXiv)

Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly.