# Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis

arXiv: https://arxiv.org/abs/2406.15796 · HTML: https://mlsanity.com/paper/2406.15796/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 22 Jun 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [Towards Safer Large Language Models through Machine Unlearning](https://mlsanity.com/paper/2402.10058/) (2024)
- [Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic](https://mlsanity.com/paper/2402.11746/) (2024)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge](https://mlsanity.com/paper/2404.05880/) (2024)
- ["According to ...": Prompting Language Models Improves Quoting from Pre-Training Data](https://mlsanity.com/paper/2305.13252/) (2023)
- [Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications](https://mlsanity.com/paper/2311.05876/) (2023)
- [Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit](https://mlsanity.com/paper/2404.01147/) (2024)

## Built on by (most cited first, 14 of 14)

- [ReLearn: Unlearning via Learning for Large Language Models](https://mlsanity.com/paper/2502.11190/) (2025)
- [Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness](https://mlsanity.com/paper/2506.05735/) (2025)
- [SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2503.14530/) (2025)
- [Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning](https://mlsanity.com/paper/2506.00876/) (2025)
- [Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation](https://mlsanity.com/paper/2505.06027/) (2025)
- [A Survey on Unlearning in Large Language Models](https://mlsanity.com/paper/2510.25117/) (2025)
- [ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging](https://mlsanity.com/paper/2503.21088/) (2025)
- [FAME: Fictional Actors for Multilingual Erasure](https://mlsanity.com/paper/2512.15235/) (2025)
- [From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models](https://mlsanity.com/paper/2604.13777/) (2026)
- [Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation](https://mlsanity.com/paper/2608.03791/) (2026)
- [UnPII: Unlearning Personally Identifiable Information with Quantifiable Exposure Risk](https://mlsanity.com/paper/2601.01786/) (2026)
- [Consistency-Aware Editing for Entity-level Unlearning in Language Models](https://mlsanity.com/paper/2601.08840/) (2025)
- [Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure](https://mlsanity.com/paper/2601.10566/) (2026)
- [Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths](https://mlsanity.com/paper/2609.34442/) (2026)

## Abstract (arXiv)

Large language model unlearning has garnered increasing attention due to its potential to address security and privacy concerns, leading to extensive research in the field. However, much of this research has concentrated on instance-level unlearning, specifically targeting the removal of predefined instances containing sensitive content. This focus has left a significant gap in the exploration of full entity-level unlearning, which is critical in real-world scenarios such as copyright protection. To this end, we propose a novel task of Entity-level unlearning, which aims to erase entity-related knowledge from the target model completely. To thoroughly investigate this task, we systematically evaluate trending unlearning algorithms, revealing that current methods struggle to achieve effective entity-level unlearning. Then, we further explore the factors that influence the performance of the unlearning algorithms, identifying that knowledge coverage and the size of the forget set play pivotal roles. Notably, our analysis also uncovers that entities introduced through fine-tuning are more vulnerable to unlearning than pre-trained entities. These findings collectively offer valuable insights for advancing entity-level unlearning for LLMs.