mlsanityData to 13 Oct 2026

Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity

arXiv 2405.13967 ↗First posted 22 May 2024cs.CLcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Recent alignment algorithms such as direct preference optimization (DPO) have been developed to improve the safety of large language models (LLMs) by training these models to match human behaviors exemplified by preference data. However, these methods are both computationally intensive and lacking in controllability and transparency, inhibiting their widespread use. Furthermore, these tuning-based methods require large-scale preference data for training and are susceptible to noisy preference data. In this paper, we introduce a tuning-free alignment alternative, ProFS (Projection Filter for Subspaces), and demonstrate its effectiveness under the use case of toxicity reduction. Grounded on theory from factor analysis, ProFS is a sample-efficient model editing approach that identifies a toxic subspace in the model parameter space and reduces model toxicity by projecting away the detected subspace. The toxic subspace is identified by extracting preference data embeddings from the language model, and removing non-toxic information from these embeddings. We show that ProFS is more sample-efficient than DPO, further showcasing greater robustness to noisy data. Finally, we attempt to connect tuning based alignment with editing, by establishing both theoretical and empirical connections between ProFS and DPO, showing that ProFS can be interpreted as a denoised version of a single DPO step.

Key facts. Cited by 34 papers in the map (data through 13 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 103 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

70 of 103 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  3. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2017Proximal Policy Optimization Algorithmscited by 14,592
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,047
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  4. 2020Learning to summarize from human feedbackcited by 2,575
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,637
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,556
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  11. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,309
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,286
  13. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,280
  14. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  15. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  16. 2022Editing Models with Task Arithmeticcited by 1,101
  17. 2022Toy Models of Superpositioncited by 950
  18. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 797
  19. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 757
  20. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 718
  21. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 706
  22. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 697
  23. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 581
  24. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 565
  25. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  26. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 529
  27. 2023Zephyr: Direct Distillation of LM Alignmentcited by 502
  28. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 437
  29. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  30. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 316
  31. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 255
  32. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  33. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 107
  34. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  35. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  36. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68
  37. 2019Attenuating Bias in Word Vectorscited by 64
  38. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 17; 2026: 12.

34 papers.

Sort:

By area:

  1. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  2. 2025Learning without training: The implicit dynamics of in-context learningcited by 42
  3. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?cited by 19
  4. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 18
  5. 2025COSMIC: Generalized Refusal Direction Identification in LLM Activationscited by 18
  6. 2025Understanding and Rectifying Safety Perception Distortion in VLMscited by 17
  7. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  8. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 11
  9. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 10
  10. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
  11. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 9
  12. 2025UniDetox: Universal Detoxification of Large Language Models via Dataset Distillationcited by 8
22 more
  1. 2025Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoderscited by 8
  2. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  3. 2024Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weightingcited by 5
  4. 2025Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editingcited by 5
  5. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppressioncited by 5
  6. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
  7. 2026Class Unlearning via Depth-Aware Removal of Forget-Specific Directionscited by 4
  8. 2025Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigationcited by 3
  9. 2025Transmuting prompts into weightscited by 3
  10. 2026CausalDetox: Causal Head Selection and Intervention for Language Model Detoxificationcited by 3
  11. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  12. 2026Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projectioncited by 2
  13. 2026OpenSafeIntent: Evaluating Intent-Calibrated Safe Completion Across Dual-Use Prompt Setscited by 2
  14. 2025Detoxification of Large Language Models through Output-layer Fusion with a Calibration Modelcited by 1
  15. 2026Do Prompts Guarantee Safety? Mitigating Toxicity from LLM Generations through Subspace Interventioncited by 1
  16. 2025A Data-Centric Approach for Safe and Secure Large Language Models against Threatening and Toxic Contentcited by 0
  17. 2025Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answerscited by 0
  18. 2026A Systematic Study of Training-Free Methods for Trustworthy Large Language Modelscited by 0
  19. 2026Toxic HallucinAItions: Perturbing Prompts and Tracing LLM Circuitscited by 0
  20. 2026QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localizationcited by 0
  21. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  22. 2026The Fragility of Trigger-Tag Mechanisms for Misuse Detection in Open-Weight LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
  2. 2025Lightweight Robust Direct Preference Optimization
  3. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  4. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
  5. 2024ROPO: Robust Preference Optimization for Large Language Models
  6. 2025Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment