# CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference

arXiv: https://arxiv.org/abs/2406.17626 · HTML: https://mlsanity.com/paper/2406.17626/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 19 of 19)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://mlsanity.com/paper/2308.13387/) (2023)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models](https://mlsanity.com/paper/2206.08325/) (2022)
- [SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures](https://mlsanity.com/paper/2110.07518/) (2021)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)

## Built on by (most cited first, 30 of 51)

- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts](https://mlsanity.com/paper/2410.10700/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs](https://mlsanity.com/paper/2410.19317/) (2024)
- [MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming](https://mlsanity.com/paper/2505.17147/) (2025)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks](https://mlsanity.com/paper/2502.11090/) (2025)
- [X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability](https://mlsanity.com/paper/2502.09990/) (2025)
- [Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks](https://mlsanity.com/paper/2503.00187/) (2025)
- [Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation](https://mlsanity.com/paper/2603.03116/) (2026)
- [Human Decision-making is Susceptible to AI-driven Manipulation](https://mlsanity.com/paper/2502.07663/) (2025)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models](https://mlsanity.com/paper/2412.11803/) (2024)
- [Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study](https://mlsanity.com/paper/2505.15389/) (2025)
- [Evaluating the Sensitivity of LLMs to Prior Context](https://mlsanity.com/paper/2506.00069/) (2025)
- [DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMs](https://mlsanity.com/paper/2602.16935/) (2026)
- [MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety](https://mlsanity.com/paper/2605.01687/) (2026)
- [AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism](https://mlsanity.com/paper/2506.03700/) (2025)
- [RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates](https://mlsanity.com/paper/2506.11083/) (2025)
- [MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?](https://mlsanity.com/paper/2507.19598/) (2025)
- [Dagger Behind Smile: Fool LLMs with a Happy Ending Story](https://mlsanity.com/paper/2501.13115/) (2025)
- [SafeMT: Multi-turn Safety for Multimodal Language Models](https://mlsanity.com/paper/2510.12133/) (2025)
- [Jailbreak Distillation: Renewable Safety Benchmarking](https://mlsanity.com/paper/2505.22037/) (2025)
- [Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection](https://mlsanity.com/paper/2510.13893/) (2025)
- [OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!](https://mlsanity.com/paper/2509.26495/) (2025)
- [ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks](https://mlsanity.com/paper/2508.16889/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models](https://mlsanity.com/paper/2601.03699/) (2026)
- [Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops](https://mlsanity.com/paper/2601.13268/) (2026)
- [SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems](https://mlsanity.com/paper/2603.17373/) (2026)

## Abstract (arXiv)

As large language models (LLMs) constantly evolve, ensuring their safety remains a critical research problem. Previous red-teaming approaches for LLM safety have primarily focused on single prompt attacks or goal hijacking. To the best of our knowledge, we are the first to study LLM safety in multi-turn dialogue coreference. We created a dataset of 1,400 questions across 14 categories, each featuring multi-turn coreference safety attacks. We then conducted detailed evaluations on five widely used open-source LLMs. The results indicated that under multi-turn coreference safety attacks, the highest attack success rate was 56% with the LLaMA2-Chat-7b model, while the lowest was 13.9% with the Mistral-7B-Instruct model. These findings highlight the safety vulnerabilities in LLMs during dialogue coreference interactions.