mlsanityData to 14 Oct 2026

Stealth Fine-Tuning: Efficiently Breaking Alignment in RVLMs Using Self-Generated CoT

arXiv 2511.14106 ↗First posted 18 Nov 2025cs.CLcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 4 more
Abstract (arXiv)

Reasoning-augmented Vision-Language Models (RVLMs) rely on safety alignment to prevent harmful behavior, yet their exposed chain-of-thought (CoT) traces introduce new attack surfaces. In this work, we find that the safety alignment of RVLMs can be easily broken through a novel attack method termed Stealth Fine-Tuning. Our method elicits harmful reasoning traces through segment-level interference and reuses the self-generated outputs as supervised fine-tuning data. To facilitate this, we introduce a turn-based weighted loss that minimizes distribution shift. In our experiment, with only 499 samples and under 3 hours on a single A100 (QLoRA), Stealth Fine-Tuning outperforms IDEATOR by 38.66% ASR while preserving general reasoning ability, as the tuned model retains the original representation distribution. Experiments on AdvBench and several general benchmarks demonstrate that Stealth Fine-Tuning is a low-cost and highly effective way to bypass alignment defenses. textcolorredDisclaimer: This paper contains content that may be disturbing or offensive.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 39 of its 43 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

39 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking18 Feb

Other references · 38

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
26 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  9. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  10. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  11. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  12. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  13. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  14. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  15. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  16. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  17. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  18. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  19. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  20. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  21. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  22. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  23. 2021Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratchcited by 102
  24. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 63
  25. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 59
  26. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselvescited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Modelscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  2. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization
  3. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  4. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning
  5. 2025PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality
  6. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
  7. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  8. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models