# Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning

arXiv: https://arxiv.org/abs/2601.03190 · HTML: https://mlsanity.com/paper/2601.03190/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 32)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning](https://mlsanity.com/paper/2404.18239/) (2024)
- [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference](https://mlsanity.com/paper/2406.08607/) (2024)
- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis](https://mlsanity.com/paper/2505.02625/) (2025)
- [Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models](https://mlsanity.com/paper/2402.05813/) (2024)
- [Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning](https://mlsanity.com/paper/2505.11953/) (2025)
- [Separable Multi-Concept Erasure from Diffusion Models](https://mlsanity.com/paper/2402.05947/) (2024)
- [Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://mlsanity.com/paper/2405.13629/) (2024)
- [PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation](https://mlsanity.com/paper/2407.05721/) (2024)
- [Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs](https://mlsanity.com/paper/2503.01090/) (2025)
- [MPO: Multilingual Safety Alignment via Reward Gap Optimization](https://mlsanity.com/paper/2505.16869/) (2025)
- [On the Limitations and Prospects of Machine Unlearning for Generative AI](https://mlsanity.com/paper/2408.00376/) (2024)
- [Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning](https://mlsanity.com/paper/2508.01181/) (2025)
- [Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models](https://mlsanity.com/paper/2506.05314/) (2025)
- [TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference](https://mlsanity.com/paper/2509.15110/) (2025)
- [Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning](https://mlsanity.com/paper/2506.00876/) (2025)
- [Not All Tokens Are Meant to Be Forgotten](https://mlsanity.com/paper/2506.03142/) (2025)
- [Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions](https://mlsanity.com/paper/2412.16838/) (2024)

## Built on by (most cited first, 10 of 10)

- [MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning](https://mlsanity.com/paper/2602.17550/) (2026)
- [BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning](https://mlsanity.com/paper/2601.09172/) (2026)
- [From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight](https://mlsanity.com/paper/2603.14389/) (2026)
- [Position: The Term "Machine Unlearning" Is Overused in LLMs](https://mlsanity.com/paper/2606.27379/) (2026)
- [VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models](https://mlsanity.com/paper/2603.18113/) (2026)
- [StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability](https://mlsanity.com/paper/2604.16385/) (2026)
- [Mitigating Error Amplification in Fast Adversarial Training](https://mlsanity.com/paper/2604.24332/) (2026)
- [Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning](https://mlsanity.com/paper/2609.05976/) (2026)
- [Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness](https://mlsanity.com/paper/2608.04519/) (2026)
- [TULIP: Targeted LLM Unlearning at Layers Identified Per-Input](https://mlsanity.com/paper/2609.34591/) (2026)

## Abstract (arXiv)

Machine unlearning aims to forget sensitive knowledge from Large Language Models (LLMs) while maintaining general utility. However, existing approaches typically treat all tokens in a response indiscriminately and enforce uncertainty over the entire vocabulary. This global treatment results in unnecessary utility degradation and extends optimization to content-agnostic regions. To address these limitations, we propose PALU (Prefix-Aware Localized Unlearning), a framework driven by a local entropy maximization objective across both temporal and vocabulary dimensions. PALU reveals that (i) suppressing the sensitive prefix alone is sufficient to sever the causal generation link, and (ii) flattening only the top-k logits is adequate to maximize uncertainty in the critical subspace. These findings allow PALU to alleviate redundant optimization across the full vocabulary and parameter space while minimizing collateral damage to general model performance. Comprehensive evaluations validate that PALU achieves superior forgetting efficacy and utility preservation compared to state-of-the-art baselines. Our code is available at https://github.com/nxZhai/PALU.