mlsanityData to 7 Oct 2026

Can Editing LLMs Inject Harm?

arXiv 2407.20224 ↗First posted 29 Jul 2024cs.CLcited by 22

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have emerged as a new information channel. Meanwhile, one critical but under-explored question is: Is it possible to bypass the safety alignment and inject harmful information into LLMs stealthily? In this paper, we propose to reformulate knowledge editing as a new type of safety threat for LLMs, namely Editing Attack, and conduct a systematic investigation with a newly constructed dataset EditAttack. Specifically, we focus on two typical safety risks of Editing Attack including Misinformation Injection and Bias Injection. For the first risk, we find that editing attacks can inject both commonsense and long-tail misinformation into LLMs, and the effectiveness for the former one is particularly high. For the second risk, we discover that not only can biased sentences be injected into LLMs with high effectiveness, but also one single biased sentence injection can degrade the overall fairness. Then, we further illustrate the high stealthiness of editing attacks. Our discoveries demonstrate the emerging misuse risks of knowledge editing techniques on compromising the safety alignment of LLMs and the feasibility of disseminating misinformation or bias with LLMs as new channels.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 107 of its 145 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

107 of 145 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Safe Harbor for AI Evaluation and Red Teaming7 Mar

Other references · 106

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
38 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  8. 2017Fake News Detection on Social Media: A Data Mining Perspectivecited by 334
  9. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  10. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  12. 2023Poisoning Language Models During Instruction Tuningcited by 254
  13. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  14. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  15. 2020Modifying Memories in Transformer Modelscited by 222
  16. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  17. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  18. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  19. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  20. 2023PMET: Precise Model Editing in a Transformercited by 170
  21. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  22. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  23. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  24. 2023On the Exploitability of Instruction Tuningcited by 110
  25. 2024The Ethics of Advanced AI Assistantscited by 103
  26. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  27. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  28. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  29. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 94
  30. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 92
  31. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  32. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  33. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  34. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  35. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  36. 2024On the Societal Impact of Open Foundation Modelscited by 64
  37. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  38. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 11; 2026: 2.

22 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  3. 2025AnyEdit: Edit Any Knowledge Encoded in Language Modelscited by 74
  4. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  5. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  6. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  7. 2024Locking Down the Finetuned LLMs Safetycited by 22
  8. 2024MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Modelscited by 20
  9. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  10. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  11. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  12. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
10 more
  1. 2024Identifying Knowledge Editing Types in Large Language Modelscited by 7
  2. 2025Tracing and Reversing Edits in LLMscited by 7
  3. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
  4. 2024Concept-ROT: Poisoning Concepts in Large Language Models with Model Editingcited by 6
  5. 2025UniErase: Towards Balanced and Precise Unlearning in Language Modelscited by 4
  6. 2025ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attackscited by 3
  7. 2025Model Editing as a Double-Edged Sword: Steering Agent Ethical Behavior Toward Beneficence or Harmcited by 2
  8. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  9. 2025How Robust is Model Editing after Fine-Tuning? An Empirical Study on Text-to-Image Diffusion Modelscited by 1
  10. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing
  2. 2025ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs
  3. 2023Fake Alignment: Are LLMs Really Aligned Well?
  4. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  5. 2025Data Security in Large Language Models: Risks, Defense, and Directions
  6. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
  7. 2024Large Language Model Safety: A Holistic Survey
  8. 2026Rendering Data Unlearnable by Exploiting LLM Alignment Mechanisms