mlsanityData to 14 Oct 2026

Red Teaming Language Model Detectors with Language Models

arXiv 2305.19713 ↗First posted 31 May 2023cs.CL cs.LGcited by 55

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. To prevent the potentially deceptive usage of LLMs, recent works have proposed algorithms to detect LLM-generated text and protect LLMs. In this paper, we investigate the robustness and reliability of these LLM detectors under adversarial attacks. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation. In both strategies, we leverage an auxiliary LLM to generate the word replacements or the instructional prompt. Different from previous works, we consider a challenging setting where the auxiliary LLM can also be protected by a detector. Experiments reveal that our attacks effectively compromise the performance of all detectors in the study with plausible generations, underscoring the urgent need to improve the robustness of LLM-generated text detection systems.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 34 of its 48 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

34 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  10. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  11. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  12. 2023A Watermark for Large Language Modelscited by 725
21 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  3. 2019ELI5: Long Form Question Answeringcited by 575
  4. 2019Defending Against Neural Fake Newscited by 559
  5. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  6. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  7. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
  8. 2018Generating Natural Language Adversarial Examplescited by 424
  9. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  10. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  11. 2021Unsolved Problems in ML Safetycited by 292
  12. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  13. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  14. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  15. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  16. 2023Protecting Language Generation Models via Invisible Watermarkingcited by 90
  17. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  18. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
  19. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 33
  20. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  21. 2020On the Robustness of Language Encoders against Grammatical Errorscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 26; 2025: 14; 2026: 5.

55 papers.

Sort:

By area:

  1. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
  2. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  3. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 171
  4. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  5. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  6. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 105
  7. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  8. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 90
  9. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  10. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 76
  11. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 55
  12. 2024SoK: Watermarking for AI-Generated Contentcited by 51
38 more
  1. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  2. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 42
  3. 2024DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarioscited by 38
  4. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  5. 2023Towards Possibilities & Impossibilities of AI-generated Text Detection: A Surveycited by 28
  6. 2024Bypassing LLM Watermarks with Color-Aware Substitutionscited by 28
  7. 2024Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attackcited by 27
  8. 2023Optimizing watermarks for large language modelscited by 25
  9. 2024Detecting AI-Generated Text: Factors Influencing Detectability with Current Methodscited by 23
  10. 2024Permute-and-Flip: An optimally stable and watermarkable decoder for LLMscited by 22
  11. 2024Locking Down the Finetuned LLMs Safetycited by 22
  12. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 21
  13. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
  14. 2023Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbotscited by 15
  15. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  16. 2025A Survey of Attacks on Large Language Modelscited by 15
  17. 2024Superficial Safety Alignment Hypothesiscited by 14
  18. 2025Understanding Teen Overreliance on AI Companion Chatbots Through Self-Reported Reddit Narrativescited by 12
  19. 2025Robustness and Cybersecurity in the EU Artificial Intelligence Actcited by 11
  20. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  21. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  22. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Modelscited by 9
  23. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 9
  24. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  25. 2024The Impact of Prompts on Zero-Shot Detection of AI-Generated Textcited by 6
  26. 2024Conversational Complexity for Assessing Risk in Large Language Modelscited by 6
  27. 2024Efficiently Identifying Watermarked Segments in Mixed-Source Textscited by 6
  28. 2024Humanizing the Machine: Proxy Attacks to Mislead LLM Detectorscited by 6
  29. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  30. 2025Encrypted Prompt: Securing LLM Applications Against Unauthorized Actionscited by 3
  31. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  32. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  33. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  34. 2025Supervision policies can shape long-term risk management in general-purpose AI modelscited by 2
  35. 2024Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?cited by 1
  36. 2026FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teamingcited by 1
  37. 2026ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluationcited by 1
  38. 2024Vietnamese AI Generated Text Detectioncited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  2. 2025Attack and defense techniques in large language models: A survey and new perspectives
  3. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks
  4. 2024Robustness of Large Language Models Against Adversarial Attacks
  5. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  7. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
  8. 2024Recent Advances in Attack and Defense Approaches of Large Language Models