# Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use

arXiv: https://arxiv.org/abs/2405.12458 · HTML: https://mlsanity.com/paper/2405.12458/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 22 of 22)

- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making](https://mlsanity.com/paper/1802.01029/) (2018)
- [Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes](https://mlsanity.com/paper/1904.09612/) (2019)
- [Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support](https://mlsanity.com/paper/2204.02310/) (2022)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services](https://mlsanity.com/paper/2303.09743/) (2023)
- [Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics](https://mlsanity.com/paper/2202.08792/) (2022)
- [Stakeholder Participation in AI: Beyond "Add Diverse Stakeholders and Stir"](https://mlsanity.com/paper/2111.01122/) (2021)
- [A Human-Centered Review of the Algorithms used within the U.S. Child Welfare System](https://mlsanity.com/paper/2003.03541/) (2020)
- [A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare](https://mlsanity.com/paper/2107.03487/) (2021)
- [Algorithms and Decision-Making in the Public Sector](https://mlsanity.com/paper/2106.03673/) (2021)
- [Investigating Human + Machine Complementarity for Recidivism Predictions](https://mlsanity.com/paper/1808.09123/) (2018)
- [Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making](https://mlsanity.com/paper/2103.02071/) (2021)
- [Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation](https://mlsanity.com/paper/2302.11623/) (2023)
- [Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems](https://mlsanity.com/paper/2001.04879/) (2020)
- [Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders](https://mlsanity.com/paper/2205.08928/) (2022)
- [The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals](https://mlsanity.com/paper/2402.18774/) (2024)
- [Towards Accountability in the Use of Artificial Intelligence for Public Administrations](https://mlsanity.com/paper/2105.01434/) (2021)
- [Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms](https://mlsanity.com/paper/2101.10367/) (2021)
- [Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?](https://mlsanity.com/paper/2109.08131/) (2021)

## Built on by (most cited first, 6 of 6)

- [Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers](https://mlsanity.com/paper/2410.22937/) (2024)
- [Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs](https://mlsanity.com/paper/2411.04994/) (2024)
- [Human-AI Narrative Synthesis to Foster Shared Understanding in Civic Decision-Making](https://mlsanity.com/paper/2509.19643/) (2025)
- [The Algorithmic Gaze of Image Quality Assessment: An Audit and Trace Ethnography of the LAION-Aesthetics Predictor](https://mlsanity.com/paper/2601.09896/) (2026)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)

## Abstract (arXiv)

As public sector agencies rapidly introduce new AI tools in high-stakes domains like social services, it becomes critical to understand how decisions to adopt these tools are made in practice. We borrow from the anthropological practice to ``study up'' those in positions of power, and reorient our study of public sector AI around those who have the power and responsibility to make decisions about the role that AI tools will play in their agency. Through semi-structured interviews and design activities with 16 agency decision-makers, we examine how decisions about AI design and adoption are influenced by their interactions with and assumptions about other actors within these agencies (e.g., frontline workers and agency leaders), as well as those above (legal systems and contracted companies), and below (impacted communities). By centering these networks of power relations, our findings shed light on how infrastructural, legal, and social factors create barriers and disincentives to the involvement of a broader range of stakeholders in decisions about AI design and adoption. Agency decision-makers desired more practical support for stakeholder involvement around public sector AI to help overcome the knowledge and power differentials they perceived between them and other stakeholders (e.g., frontline workers and impacted community members). Building on these findings, we discuss implications for future research and policy around actualizing participatory AI approaches in public sector contexts.