mlsanityData to 7 Oct 2026

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

arXiv 2401.01967 ↗First posted 3 Jan 2024cs.CL cs.AIcited by 179

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

While alignment algorithms are now commonly used to tune pre-trained language models towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained language model, GPT2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 45 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

42 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
26 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  10. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  13. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  14. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  15. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  16. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  17. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  18. 2023In-Context Learning Creates Task Vectorscited by 302
  19. 2023Function Vectors in Large Language Modelscited by 292
  20. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  21. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  22. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  23. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  24. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  25. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  26. 2022The CRINGE Loss: Learning what language not to modelcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 65; 2026: 43.

179 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  7. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  8. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  11. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  12. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
38 more
  1. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  2. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  3. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  4. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  5. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  6. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  7. 2025Persona Features Control Emergent Misalignmentcited by 76
  8. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  9. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  10. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2024ICLR: In-Context Learning of Representationscited by 49
  13. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  14. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  15. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  16. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  17. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  18. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  19. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  20. 2025A Survey of Direct Preference Optimizationcited by 35
  21. 2024Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Modelscited by 34
  22. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  23. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  24. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  25. 2024Immunization against harmful fine-tuning attackscited by 33
  26. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  27. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  28. 2024Towards Reliable Evaluation of Behavior Steering Interventions in LLMscited by 33
  29. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  30. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  31. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  32. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  33. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 27
  34. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  35. 2024Dissecting Fine-Tuning Unlearning in Large Language Modelscited by 25
  36. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 24
  37. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  38. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23

50 most cited of 179 All 179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
  2. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think
  3. 2024Direct Preference Optimization with an Offset
  4. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  5. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  6. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
  7. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
  8. 2025What Matters in Data for DPO?