mlsanityData to 7 Oct 2026

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

arXiv 2407.08770 ↗First posted 11 Jul 2024cs.AIcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 66 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

47 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languages23 Jun
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
31 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2022Mass-Editing Memory in a Transformercited by 976
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  10. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  13. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  14. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  15. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  16. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  18. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  19. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  20. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  21. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  22. 2023Mitigating the Alignment Tax of RLHFcited by 181
  23. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  24. 2018Challenges for Toxic Comment Classification: An In-Depth Error Analysiscited by 55
  25. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  26. 2023Language Model Alignment with Elastic Resetcited by 34
  27. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  28. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuningcited by 24
  29. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  30. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  31. 2024Parameter-Efficient Detoxification with Contrastive Decodingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  3. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  5. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 29
  6. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  7. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 17
  8. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  9. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMscited by 9
  10. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  11. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  12. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
11 more
  1. 2025Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detectioncited by 6
  2. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  3. 2026Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablationcited by 2
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  5. 2024Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Modelscited by 1
  6. 2025N-GLARE: An Non-Generative Latent Representation-Efficient LLM Safety Evaluatorcited by 1
  7. 2025Decomposing Task Vectors for Refined Model Editingcited by 1
  8. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  9. 2026Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignmentcited by 0
  10. 2026How LLMs Might Thinkcited by 0
  11. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Detoxifying Large Language Models via Knowledge Editing
  2. 2025Command-V: Pasting LLM Behaviors via Activation Profiles
  3. 2026An Efficient and Modular Framework for Targeted Harm Mitigation in LLMS
  4. 2026Detoxifying LLMs via Representation Erasure-Based Preference Optimization
  5. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
  6. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  7. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMs
  8. 2024PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach