# Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

arXiv: https://arxiv.org/abs/2405.09679 · HTML: https://mlsanity.com/paper/2405.09679/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 15 May 2024.

## Builds on (most cited first, 19 of 19)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Can large language models build causal graphs?](https://mlsanity.com/paper/2303.05279/) (2023)
- [Causal Discovery with Language Models as Imperfect Experts](https://mlsanity.com/paper/2307.02390/) (2023)
- [Causal-Discovery Performance of ChatGPT in the context of Neuropathic Pain Diagnosis](https://mlsanity.com/paper/2301.13819/) (2023)
- [AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms](https://mlsanity.com/paper/2306.03280/) (2023)
- [Global AI Ethics: A Review of the Social Impacts and Ethical Implications of Artificial Intelligence](https://mlsanity.com/paper/1907.07892/) (2019)
- [Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning](https://mlsanity.com/paper/2401.00139/) (2023)
- [Crowdsourcing Impacts: Exploring the Utility of Crowds for Anticipating Societal Impacts of Algorithmic Decision Making](https://mlsanity.com/paper/2207.09525/) (2022)
- [Anticipatory Ethics and the Role of Uncertainty](https://mlsanity.com/paper/2011.13170/) (2020)
- [Auditing Recommender Systems -- Putting the DSA into practice with a risk-scenario-based approach](https://mlsanity.com/paper/2302.04556/) (2023)
- [My Future with My Chatbot: A Scenario-Driven, User-Centric Approach to Anticipating AI Impacts](https://mlsanity.com/paper/2401.14533/) (2024)

## Built on by (most cited first, 5 of 5)

- [Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Making](https://mlsanity.com/paper/2502.14869/) (2025)
- [Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Society](https://mlsanity.com/paper/2506.05605/) (2025)
- [Synthia: Scalable Grounded Persona Generation from Social Media Data](https://mlsanity.com/paper/2507.14922/) (2025)
- [Informing AI Policy Assessment using Large-Scale Simulation of Interventions](https://mlsanity.com/paper/2605.27395/) (2026)
- [GPS-Bench: A Governance Policy Benchmark for Automating Policy Analysis](https://mlsanity.com/paper/2609.03553/) (2026)

## Abstract (arXiv)

The rapid advancement of AI technologies yields numerous future impacts on individuals and society. Policymakers are tasked to react quickly and establish policies that mitigate those impacts. However, anticipating the effectiveness of policies is a difficult task, as some impacts might only be observable in the future and respective policies might not be applicable to the future development of AI. In this work we develop a method for using large language models (LLMs) to evaluate the efficacy of a given piece of policy at mitigating specified negative impacts. We do so by using GPT-4 to generate scenarios both pre- and post-introduction of policy and translating these vivid stories into metrics based on human perceptions of impacts. We leverage an already established taxonomy of impacts of generative AI in the media environment to generate a set of scenario pairs both mitigated and non-mitigated by the transparency policy in Article 50 of the EU AI Act. We then run a user study (n=234) to evaluate these scenarios across four risk-assessment dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. We find that this transparency legislation is perceived to be effective at mitigating harms in areas such as labor and well-being, but largely ineffective in areas such as social cohesion and security. Through this case study we demonstrate the efficacy of our method as a tool to iterate on the effectiveness of policy for mitigating various negative impacts. We expect this method to be useful to researchers or other stakeholders who want to brainstorm the potential utility of different pieces of policy or other mitigation strategies.