How User-AI Mistreatment Occurs and Matters in Conversational Systems?
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Safety research often focuses on model-generated harms, but users may also direct hostility, coercion, and adversarial pressure at models. Understanding how and when that occurs is essential for accurately interpreting model behaviour, alignment drift, and real-world deployment risks. In this paper, we audit 777K English LMSYS-Chat-1M conversations with two independent detectors: an eight-category lexicon for hostility directed at the model, and the dataset's moderation signal; and show that they capture different, weakly overlapping phenomena. The lexicon identifies insults, threats, and jailbreak coercion aimed at the assistant, while moderation flags are dominated by toxic-content solicitation rather than hostility at the model. Together, they mark about 5% of user turns; adjusting the narrower lexicon-harassment union for measured precision puts mistreatment aimed at the assistant at 0.90%. These absolute rates describe arena-style evaluation traffic and should not be read as deployment-wide base rates. We find that user hostility varies 13-fold across models, driven largely by who each model attracts rather than by model behaviour: first-turn hostility spreads far wider than post-response hostility, and more than fifteenfold separates the extremes even after deduplicating opening prompts. Within conversations, assistant apologies are consistently associated with higher odds of next-turn hostility under both detectors; the effect survives restricting to non-refused prior turns and to jailbreak-free conversations, and is positive in 20 of 23 models. Yet across models, more apologetic models receive less hostility overall. Finally, hostility also shows temporal structure, with coercive openings front-loading the first turn while affective hostility accumulates over a session. We release the lexicon, the detector cross-validation pipeline, and all derived tables.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 12 of its 47 references that are in the map. First posted on arXiv 11 Sep 2026.
Builds on
12 of 47 references.
References · 12
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
- 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
- 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
- 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
- 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Refusal in Language Models Is Mediated by a Single Direction
- 2026Investigating and Alleviating Harm Amplification in LLM Interactions
- 2026When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents
- 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models
- 2024Annotation alignment: Comparing LLM and human annotations of conversational safety
- 2026EUDAIMONIA: Evaluating Undesirable Dynamics in AI
- 2026Evaluating Language Model Safety Across Long Adversarial Conversations
- 2026No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios