Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm. We term this Conversational Risk Accumulation (CRA): gradual intent drift, fragmented assembly of prohibited instructions, and sensitivity build-up from repeated disclosures. We propose a session-layer CRA Framework that tracks three trajectory signals: semantic drift from a session anchor, a sensitivity-weighted information accumulation graph over extracted entities, and a compliance-gradient signal capturing increasing willingness to comply. For scoring, we provide (i) an unsupervised convex fusion for attribution and ablations, and (ii) CRA-Net DA, a compact learned trajectory model trained with family-adversarial objectives to reduce length and topic-coverage confounds. To benchmark CRA, we release CRA-Bench v0.1 (1,200 eight-turn sessions across three threat families with topic-matched benign twins), CRA-Bench v0.2 (LLM-paraphrased variants to reduce template artifacts), and an extended 5-family set (2,000 sessions adding persona priming and context stuffing). We introduce a trajectory-native evaluation protocol with session-level splits, mixed-set threshold calibration, Trajectory AUROC, turns-to-detection, calibrated false-positive metrics, bootstrap confidence intervals, leave-one-family-out diagnostic stress tests, and synthetic-to-human transfer checks. Claims focus on within-distribution session scoring on CRA-Bench and human-transfer subsets.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 18 of its 32 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
18 of 32 references.
Closest ancestors · 2
Same direction, newest first
- 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories8 Apr
- 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference25 Jun
Other references · 16
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
4 more
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
- 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
- 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
- 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Securitycited by 41
Built on by
Not cited yet.