mlsanityData to 14 Oct 2026

How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

arXiv 2601.06033 ↗First posted 10 Nov 2025cs.HC cs.CR cs.CYcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of attacks, lowering the barrier to entry for creating harmful content, including sophisticated propaganda and deepfakes. Conversely, defenders envision leveraging GenAI to detect and mitigate harmful content at scale, conduct investigations, deploy persuasive counternarratives, improve moderator wellbeing, and offer user support. This work provides a strategic framework for understanding GenAI's impact on Trust & Safety and charts a path for its responsible use in creating safer online environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 33 of its 171 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

33 of 171 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
  7. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 193
  8. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  9. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
  10. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  11. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 67
  12. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 63
21 more
  1. 2020Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practicescited by 39
  2. 2017Like trainer, like bot? Inheritance of bias in algorithmic content moderationcited by 35
  3. 2023How Knowledge Workers Think Generative AI Will (Not) Transform Their Industriescited by 30
  4. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 27
  5. 2023Concrete Problems in AI Safety, Revisitedcited by 22
  6. 2023Safety and Fairness for Content Moderation in Generative Modelscited by 16
  7. 2022A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Researchcited by 14
  8. 2023Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholderscited by 13
  9. 2024Analyzing the AI Nudification Application Ecosystemcited by 10
  10. 2024An Explorative Study of Pig Butchering Scamscited by 5
  11. 2022What does it mean to be a responsible AI practitioner: An ontology of roles and skillscited by 4
  12. 2024Content Moderation Justice and Fairness on Social Media: Comparisons Across Different Contexts and Platformscited by 4
  13. 2024Do Responsible AI Artifacts Advance Stakeholder Goals? Four Key Barriers Perceived by Legal and Civil Stakeholderscited by 4
  14. 2024Exploring the Boundaries of Content Moderation in Text-to-Image Generationcited by 4
  15. 2023SoK: Safer Digital-Safety Research Involving At-Risk Userscited by 3
  16. 2024Debiasing Text Safety Classifiers through a Fairness-Aware Ensemblecited by 3
  17. 2025"Hello, is this Anna?": Unpacking the Lifecycle of Pig-Butchering Scamscited by 2
  18. 2022Listen to what they say: Better understand and detect online misinformation with user feedbackcited by 1
  19. 2024Give and Take: An End-To-End Investigation of Giveaway Scam Conversion Ratescited by 1
  20. 2024Envisioning New Futures of Positive Social Technology: Beyond Paradigms of Fixing, Protecting, and Preventingcited by 1
  21. 2025Design for Hope: Cultivating Deliberate Hope in the Face of Complex Societal Challengescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Review of Generative AI Methods in Cybersecurity
  2. 2023Identifying and Mitigating the Security Risks of Generative AI
  3. 2025Jailbreaking Generative AI: Multivector Phishing Threats and Transformer based Defenses
  4. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
  5. 2025GenAI Security: Outsmarting the Bots with a Proactive Testing Framework
  6. 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective
  7. 2026From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI
  8. 2023Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Models