mlsanityData to 7 Oct 2026

SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics

arXiv 2604.16358 ↗First posted 18 Mar 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

MLLMs are increasingly deployed in multi-turn settings, where attackers can escalate unsafe intent through the evolving visual-text history and exploit long-context safety decay. Yet safety alignment is still dominated by single-turn data and fixed-template dialogues, leaving a mismatch between training and deployment. To bridge this gap, we propose SaFeR-Steer, a progressive multi-turn alignment framework that combines staged synthetic bootstrapping with tutor-in-the-loop GRPO to train a single student under adaptive, on-policy attacks. We also introduce Trajectory-Consistent Summative Reward (TCSR), which aggregates the historical minimum and average of turn rewards so that any low-quality turn affects the trajectory-level return. I. Dataset. We release STEER, a multi-turn multimodal safety dataset with STEER-SFT (12,934), STEER-RL (2,000), and STEER-Bench (3,227) dialogues spanning 1-10 turns. II. Experiment. Starting from Qwen2.5-VL-3B/7B, SaFeR-Steer substantially improves Safety/Helpfulness on both single-turn (48.30/45.86 rightarrow 81.84/70.77 for 3B; 56.21/60.32 rightarrow 87.89/77.40 for 7B) and multi-turn benchmarks (12.55/27.13 rightarrow 55.58/70.27 for 3B; 24.66/46.48 rightarrow 64.89/72.35 for 7B), shifting failures to later turns and yielding robustness beyond scaling alone. Code is available at https://github.com/Ed-Bg/SaFeR-Steer-full

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 109 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

38 of 109 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model10 May
  2. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback22 Mar
  3. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model17 Jun
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  8. 2025OpenAI GPT-5 System Cardcited by 910
  9. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  10. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  11. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  12. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
21 more
  1. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 208
  2. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  4. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  5. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  6. 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrievalcited by 65
  7. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialoguecited by 64
  8. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 59
  9. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  10. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
  11. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  12. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  13. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  14. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  15. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Modelscited by 12
  16. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 8
  17. 2025Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embeddingcited by 5
  18. 2025SafeMT: Multi-turn Safety for Multimodal Language Modelscited by 4
  19. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  20. 2024Attacking Transformers with Feature Diversity Adversarial Perturbationcited by 3
  21. 2025Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Messagecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures
  2. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models
  3. 2026Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment
  4. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
  5. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents
  6. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences
  7. 2026AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation
  8. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization