mlsanityData to 7 Oct 2026

Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models

arXiv 2401.10647 ↗First posted 19 Jan 2024cs.CLcited by 20

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

In the rapidly advancing field of artificial intelligence, the concept of Red-Teaming or Jailbreaking large language models (LLMs) has emerged as a crucial area of study. This approach is especially significant in terms of assessing and enhancing the safety and robustness of these models. This paper investigates the intricate consequences of such modifications through model editing, uncovering a complex relationship between enhancing model accuracy and preserving its ethical integrity. Our in-depth analysis reveals a striking paradox: while injecting accurate information is crucial for model reliability, it can paradoxically destabilize the model's foundational framework, resulting in unpredictable and potentially unsafe behaviors. Additionally, we propose a benchmark dataset NicheHazardQA to investigate this unsafe behavior both within the same and cross topical domain. This aspect of our research sheds light on how the edits, impact the model's safety metrics and guardrails. Our findings show that model editing serves as a cost-effective tool for topical red-teaming by methodically applying targeted edits and evaluating the resultant model behavior.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 24 of its 35 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug

Other references · 23

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  6. 2021Editing Factual Knowledge in Language Modelscited by 548
  7. 2023Erasing Concepts from Diffusion Modelscited by 530
  8. 2022Memory-Based Model Editing at Scalecited by 443
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  10. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  11. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  12. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
11 more
  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  6. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  7. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  8. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  9. 2023Unveiling the Pitfalls of Knowledge Editing for Large Language Modelscited by 43
  10. 2023Evaluating the Reversal Curse in Model Editingcited by 34
  11. 2023DocTER: Evaluating Document-based Knowledge Editingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 7; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  2. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  3. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  4. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  5. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  6. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 22
  7. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  8. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  9. 2024How to Make LLMs Forget: On Reversing In-Context Knowledge Editscited by 9
  10. 2025Revealing and Mitigating Over-Attention in Knowledge Editingcited by 8
  11. 2025Tracing and Reversing Edits in LLMscited by 7
  12. 2025Attributional Safety Failures in Large Language Models under Code-Mixed Perturbationscited by 6
8 more
  1. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  2. 2024"Flex Tape Can't Fix That": Bias and Misinformation in Edited Language Modelscited by 4
  3. 2024Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performancecited by 4
  4. 2024Composable Interventions for Language Modelscited by 4
  5. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  6. 2025Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questionscited by 1
  7. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  8. 2026Selective Knowledge Edit Reversal via Gated Singular Vector Shrinkagecited by 0