# Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs

arXiv: https://arxiv.org/abs/2411.04994 · HTML: https://mlsanity.com/paper/2411.04994/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 143 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 24 of 24)

- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making](https://mlsanity.com/paper/1802.01029/) (2018)
- [Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem](https://mlsanity.com/paper/2310.02521/) (2023)
- [Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors](https://mlsanity.com/paper/2105.02980/) (2021)
- [Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support](https://mlsanity.com/paper/2204.02310/) (2022)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services](https://mlsanity.com/paper/2303.09743/) (2023)
- [Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics](https://mlsanity.com/paper/2202.08792/) (2022)
- [Overcoming Failures of Imagination in AI Infused System Development and Deployment](https://mlsanity.com/paper/2011.13416/) (2020)
- [Algorithms and Decision-Making in the Public Sector](https://mlsanity.com/paper/2106.03673/) (2021)
- [Auditing Work: Exploring the New York City algorithmic bias audit regime](https://mlsanity.com/paper/2402.08101/) (2024)
- [AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries](https://mlsanity.com/paper/2105.10356/) (2021)
- [Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders](https://mlsanity.com/paper/2205.08928/) (2022)
- [The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals](https://mlsanity.com/paper/2402.18774/) (2024)
- [Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms](https://mlsanity.com/paper/2101.10367/) (2021)
- [Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge](https://mlsanity.com/paper/2308.15700/) (2023)
- [Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use](https://mlsanity.com/paper/2405.12458/) (2024)
- [Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool](https://mlsanity.com/paper/2405.07715/) (2024)
- [The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment](https://mlsanity.com/paper/2404.13802/) (2024)
- [Ghosting the Machine: Judicial Resistance to a Recidivism Risk Assessment Instrument](https://mlsanity.com/paper/2306.06573/) (2023)
- [AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance](https://mlsanity.com/paper/2404.14660/) (2024)
- [Public Technologies Transforming Work of the Public and the Public Sector](https://mlsanity.com/paper/2405.18579/) (2024)

## Built on by (most cited first, 3 of 3)

- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Now You See Me: Designing Responsible AI Dashboards for Early-Stage Health Innovation](https://mlsanity.com/paper/2602.23378/) (2026)

## Abstract (arXiv)

Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. In this paper, we conduct the first empirical study of how United States cities' procurement practices shape critical decisions surrounding public sector AI. We conduct semi-structured interviews with 19 city employees who oversee AI procurement across 7 U.S. cities. We found that cities' legacy procurement practices, which are shaped by decades-old laws and norms, establish infrastructure that determines which AI is purchased, and which actors hold decision-making power over procured AI. We characterize the emerging actions cities have taken to adapt their purchasing practices to address algorithmic harms. From employees' reflections on real-world AI procurements, we identify three key challenges that motivate but are not fully addressed by existing AI procurement reform initiatives. Based on these findings, we discuss implications and opportunities for the FAccT community to support cities in foreseeing and preventing AI harms throughout the public procurement processes.