mlsanityData to 14 Oct 2026

Toxicity in ChatGPT: Analyzing Persona-assigned Language Models

arXiv 2304.05335 ↗First posted 11 Apr 2023cs.CL cs.AI cs.LGcited by 360

Not on the map. Why?

Cited by 62 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3x more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.

Key facts. Cited by 360 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

32 of 52 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
20 more
  1. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  2. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  5. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  6. 2019Measuring Bias in Contextualized Word Representationscited by 276
  7. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 270
  8. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  9. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  10. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  11. 2018Learning Gender-Neutral Word Embeddingscited by 169
  12. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  13. 2019Does Object Recognition Work for Everyone?cited by 151
  14. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  15. 2020Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddingscited by 79
  16. 2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 51
  17. 2017Detecting Hate Speech in Social Mediacited by 42
  18. 2022Leashing the Inner Demons: Self-Detoxification for Language Modelscited by 19
  19. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Modelscited by 18
  20. 2022Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 82; 2024: 133; 2025: 92; 2026: 53.

360 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  9. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  12. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
38 more
  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  4. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  5. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  6. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  8. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  9. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 199
  10. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  11. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  12. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  13. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  14. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  15. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  16. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  17. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  18. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  19. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  20. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  21. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  22. 2023On the Exploitability of Instruction Tuningcited by 116
  23. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  24. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
  25. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  26. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  27. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  28. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  29. 2024Are self-explanations from Large Language Models faithful?cited by 101
  30. 2023Aligning Language Models to User Opinionscited by 99
  31. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  32. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  33. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  34. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  35. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  36. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  37. 2025Persona Features Control Emergent Misalignmentcited by 86
  38. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85

50 most cited of 360 All 360 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Stars, Stripes, and Silicon: Unravelling the ChatGPT's All-American, Monochrome, Cis-centric Bias
  2. 2026Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study
  3. 2026Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensembling
  4. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?
  5. 2026Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining
  6. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  7. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
  8. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media