Risks of AI-driven product development and strategies for their mitigation
Not on the map. Why?
Abstract (arXiv)
Humanity is progressing towards automated product development, a trend that promises faster creation of better products and thus the acceleration of technological progress. However, increasing reliance on non-human agents for this process introduces many risks. This perspective aims to initiate a discussion on these risks and appropriate mitigation strategies. To this end, we outline a set of principles for safer AI-driven product development which emphasize human oversight, accountability, and explainable design, among others. The risk assessment covers both technical risks which affect product quality and safety, and sociotechnical risks which affect society. While AI-driven product development is still in its early stages, this discussion will help balance its opportunities and risks without delaying essential progress in understanding, norm-setting, and regulation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 82 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
38 of 82 references.
References · 38
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2016Deep Learning with Differential Privacycited by 2,728
- 2018Datasheets for Datasetscited by 1,386
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2018Model Cards for Model Reportingcited by 1,080
- 2019Attention is not Explanationcited by 929
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
- 2023A Watermark for Large Language Modelscited by 725
- 2021Carbon Emissions and Large Neural Network Trainingcited by 567
26 more
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
- 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 165
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
- 2025Measuring AI Ability to Complete Long Software Taskscited by 156
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
- 2023Auditing large language models: a three-layered approachcited by 137
- 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
- 2023AI model GPT-3 (dis)informs us better than humanscited by 78
- 2018Motivating the Rules of the Game for Adversarial Example Researchcited by 57
- 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 48
- 2023Testing Language Model Agents Safely in the Wildcited by 46
- 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 32
- 2024Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concernscited by 28
- 2023Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Studycited by 26
- 2024From Insights to Actions: The Impact of Interpretability and Analysis Research on NLPcited by 20
- 2024Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitationcited by 19
- 2022Risk management in the Artificial Intelligence Actcited by 16
- 2021Why and How Governments Should Monitor AI Developmentcited by 14
- 2025AI Safety for Everyonecited by 14
- 2023Opportunities for Large Language Models and Discourse in Engineering Designcited by 6
- 2024Generative Optimization: A Perspective on AI-Enhanced Problem Solving in Engineeringcited by 5
- 2023To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processingcited by 4
- 2024Deep Generative Model for Mechanical System Configuration Designcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.