mlsanityData to 7 Oct 2026

Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AI

arXiv 2507.05660 ↗First posted 8 Jul 2025cs.CR cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Customizing Large Language Models (LLMs) on untrusted datasets poses severe risks of injecting toxic behaviors. In this work, we introduce Optimus, a novel defense framework designed to mitigate fine-tuning harms while preserving conversational utility. Unlike existing defenses that rely heavily on precise toxicity detection or restrictive filtering, Optimus addresses the critical challenge of ensuring robust mitigation even when toxicity classifiers are imperfect or biased. Optimus integrates a training-free toxicity classification scheme that repurposes the safety alignment of commodity LLMs, and employs a dual-strategy alignment process combining synthetic "healing data" with Direct Preference Optimization (DPO) to efficiently steer models toward safety. Extensive evaluations demonstrate that Optimus mitigates toxicity even when relying on extremely biased classifiers (with up to 85% degradation in Recall). Optimus outperforms the state-of-the-art defense StarDSS and exhibits strong resilience against adaptive adversarial and jailbreak attacks. Our source code and datasets are available at https://github.com/secml-lab-vt/Optimus

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 47 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

30 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Shape it Up! Restoring LLM Safety during Finetuning22 May
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  3. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  5. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  7. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  12. 2020Recipes for building an open-domain chatbotcited by 593
11 more
  1. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  2. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  3. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  4. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  5. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  6. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  7. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 46
  8. 2023You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Contentcited by 37
  9. 2021Assessing Political Prudence of Open-domain Chatbotscited by 20
  10. 2021Assessing Dialogue Systems with Distribution Distancescited by 16
  11. 2024AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attemptscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Conversational Agents against Imperceptible Toxicity Triggers
  2. 2025ToxSearch: Evolving Prompts for Toxicity Search in Large Language Models
  3. 2026Detoxifying LLMs via Representation Erasure-Based Preference Optimization
  4. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation
  5. 2024Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents
  6. 2025Probing AI Safety with Source Code
  7. 2026Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
  8. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions