# Co-designing an AI Impact Assessment Report Template with AI Practitioners and AI Compliance Experts

arXiv: https://arxiv.org/abs/2407.17374 · HTML: https://mlsanity.com/paper/2407.17374/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 56 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency](https://mlsanity.com/paper/2001.09734/) (2020)
- [AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms](https://mlsanity.com/paper/2306.03280/) (2023)
- [Principles to Practices for Responsible AI: Closing the Gap](https://mlsanity.com/paper/2006.04707/) (2020)
- [Use case cards: a use case reporting framework inspired by the European AI Act](https://mlsanity.com/paper/2306.13701/) (2023)
- [ExploreGen: Large Language Models for Envisioning the Uses and Risks of AI Technologies](https://mlsanity.com/paper/2407.12454/) (2024)
- [Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits](https://mlsanity.com/paper/2403.00145/) (2024)
- [ECCOLA -- a Method for Implementing Ethically Aligned AI Systems](https://mlsanity.com/paper/2004.08377/) (2020)
- [Crowdsourcing Impacts: Exploring the Utility of Crowds for Anticipating Societal Impacts of Algorithmic Decision Making](https://mlsanity.com/paper/2207.09525/) (2022)
- [RAI Guidelines: Method for Generating Responsible AI Guidelines Grounded in Regulations and Usable by (Non-)Technical Roles](https://mlsanity.com/paper/2307.15158/) (2023)
- [AI Risk Profiles: A Standards Proposal for Pre-Deployment AI Risk Disclosures](https://mlsanity.com/paper/2309.13176/) (2023)
- [My Model is Unfair, Do People Even Care? Visual Design Affects Trust and Perceived Bias in Machine Learning](https://mlsanity.com/paper/2308.03299/) (2023)
- [Good Intentions, Risky Inventions: A Method for Assessing the Risks and Benefits of AI in Mobile and Wearable Uses](https://mlsanity.com/paper/2407.09322/) (2024)

## Built on by (most cited first, 10 of 10)

- [From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems](https://mlsanity.com/paper/2502.07974/) (2025)
- [Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models](https://mlsanity.com/paper/2508.16673/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [Privy: Envisioning and Mitigating Privacy Risks for Consumer-facing AI Product Concepts](https://mlsanity.com/paper/2509.23525/) (2025)
- [The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms](https://mlsanity.com/paper/2508.16672/) (2025)
- [How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?](https://mlsanity.com/paper/2602.18464/) (2026)
- [The Hall of AI Fears and Hopes: Comparing the Views of AI Influencers and those of Members of the U.S. Public Through an Interactive Platform](https://mlsanity.com/paper/2504.06016/) (2025)
- [Agent-Supported Foresight for AI Systemic Risks: AI Agents for Breadth, Experts for Judgment](https://mlsanity.com/paper/2602.08565/) (2026)
- [Engaged AI Governance: Addressing the Last Mile Challenge Through Internal Expert Collaboration](https://mlsanity.com/paper/2604.21554/) (2026)
- [What We Know about Responsible AI Practices in Industry: A Half Decade of Empirical Research](https://mlsanity.com/paper/2608.10431/) (2026)

## Abstract (arXiv)

In the evolving landscape of AI regulation, it is crucial for companies to conduct impact assessments and document their compliance through comprehensive reports. However, current reports lack grounding in regulations and often focus on specific aspects like privacy in relation to AI systems, without addressing the real-world uses of these systems. Moreover, there is no systematic effort to design and evaluate these reports with both AI practitioners and AI compliance experts. To address this gap, we conducted an iterative co-design process with 14 AI practitioners and 6 AI compliance experts and proposed a template for impact assessment reports grounded in the EU AI Act, NIST's AI Risk Management Framework, and ISO 42001 AI Management System. We evaluated the template by producing an impact assessment report for an AI-based meeting companion at a major tech company. A user study with 8 AI practitioners from the same company and 5 AI compliance experts from industry and academia revealed that our template effectively provides necessary information for impact assessments and documents the broad impacts of AI systems. Participants envisioned using the template not only at the pre-deployment stage for compliance but also as a tool to guide the design stage of AI uses.