# Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning

arXiv: https://arxiv.org/abs/1907.11274 · HTML: https://mlsanity.com/paper/1907.11274/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2019.

## Built on by (most cited first, 12 of 12)

- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [The Gradient of Generative AI Release: Methods and Considerations](https://mlsanity.com/paper/2302.04844/) (2023)
- [Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models](https://mlsanity.com/paper/2211.14946/) (2022)
- [Democratising AI: Multiple Meanings, Goals, and Methods](https://mlsanity.com/paper/2303.12642/) (2023)
- [Towards Responsible Natural Language Annotation for the Varieties of Arabic](https://mlsanity.com/paper/2203.09597/) (2022)
- [Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers](https://mlsanity.com/paper/2104.01604/) (2021)
- [The Deepfake Detection Dilemma: A Multistakeholder Exploration of Adversarial Dynamics in Synthetic Media](https://mlsanity.com/paper/2102.06109/) (2021)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks](https://mlsanity.com/paper/2603.01291/) (2026)
- [Rethinking Text-to-Image as Semantic-Aware Data Augmentation for Indoor Scene Recognition](https://mlsanity.com/paper/2606.18555/) (2026)

## Abstract (arXiv)

The aim of this paper is to facilitate nuanced discussion around research norms and practices to mitigate the harmful impacts of advances in machine learning (ML). We focus particularly on the use of ML to create "synthetic media" (e.g. to generate or manipulate audio, video, images, and text), and the question of what publication and release processes around such research might look like, though many of the considerations discussed will apply to ML research more broadly. We are not arguing for any specific approach on when or how research should be distributed, but instead try to lay out some useful tools, analogies, and options for thinking about these issues. We begin with some background on the idea that ML research might be misused in harmful ways, and why advances in synthetic media, in particular, are raising concerns. We then outline in more detail some of the different paths to harm from ML research, before reviewing research risk mitigation strategies in other fields and identifying components that seem most worth emulating in the ML and synthetic media research communities. Next, we outline some important dimensions of disagreement on these issues which risk polarizing conversations. Finally, we conclude with recommendations, suggesting that the machine learning community might benefit from: working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.