# How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

arXiv: https://arxiv.org/abs/2601.06033 · HTML: https://mlsanity.com/paper/2601.06033/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 33 of its 171 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021)
- [Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices](https://mlsanity.com/paper/2006.12358/) (2020)
- [Like trainer, like bot? Inheritance of bias in algorithmic content moderation](https://mlsanity.com/paper/1707.01477/) (2017)
- [How Knowledge Workers Think Generative AI Will (Not) Transform Their Industries](https://mlsanity.com/paper/2310.06778/) (2023)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [Concrete Problems in AI Safety, Revisited](https://mlsanity.com/paper/2401.10899/) (2023)
- [Safety and Fairness for Content Moderation in Generative Models](https://mlsanity.com/paper/2306.06135/) (2023)
- [A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Research](https://mlsanity.com/paper/2207.04171/) (2022)
- [Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders](https://mlsanity.com/paper/2303.06794/) (2023)
- [Analyzing the AI Nudification Application Ecosystem](https://mlsanity.com/paper/2411.09751/) (2024)
- [An Explorative Study of Pig Butchering Scams](https://mlsanity.com/paper/2412.15423/) (2024)
- [What does it mean to be a responsible AI practitioner: An ontology of roles and skills](https://mlsanity.com/paper/2205.03946/) (2022)
- [Content Moderation Justice and Fairness on Social Media: Comparisons Across Different Contexts and Platforms](https://mlsanity.com/paper/2403.06034/) (2024)
- [Do Responsible AI Artifacts Advance Stakeholder Goals? Four Key Barriers Perceived by Legal and Civil Stakeholders](https://mlsanity.com/paper/2408.12047/) (2024)
- [Exploring the Boundaries of Content Moderation in Text-to-Image Generation](https://mlsanity.com/paper/2409.17155/) (2024)
- [SoK: Safer Digital-Safety Research Involving At-Risk Users](https://mlsanity.com/paper/2309.00735/) (2023)
- [Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble](https://mlsanity.com/paper/2409.13705/) (2024)
- ["Hello, is this Anna?": Unpacking the Lifecycle of Pig-Butchering Scams](https://mlsanity.com/paper/2503.20821/) (2025)
- [Listen to what they say: Better understand and detect online misinformation with user feedback](https://mlsanity.com/paper/2210.17166/) (2022)

## Abstract (arXiv)

Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of attacks, lowering the barrier to entry for creating harmful content, including sophisticated propaganda and deepfakes. Conversely, defenders envision leveraging GenAI to detect and mitigate harmful content at scale, conduct investigations, deploy persuasive counternarratives, improve moderator wellbeing, and offer user support. This work provides a strategic framework for understanding GenAI's impact on Trust & Safety and charts a path for its responsible use in creating safer online environments.