mlsanityData to 14 Oct 2026

Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning

arXiv 2601.03190 ↗First posted 6 Jan 2026cs.CLcited by 10

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning aims to forget sensitive knowledge from Large Language Models (LLMs) while maintaining general utility. However, existing approaches typically treat all tokens in a response indiscriminately and enforce uncertainty over the entire vocabulary. This global treatment results in unnecessary utility degradation and extends optimization to content-agnostic regions. To address these limitations, we propose PALU (Prefix-Aware Localized Unlearning), a framework driven by a local entropy maximization objective across both temporal and vocabulary dimensions. PALU reveals that (i) suppressing the sensitive prefix alone is sufficient to sever the causal generation link, and (ii) flattening only the top-k logits is adequate to maximize uncertainty in the critical subspace. These findings allow PALU to alleviate redundant optimization across the full vocabulary and parameter space while minimizing collateral damage to general model performance. Comprehensive evaluations validate that PALU achieves superior forgetting efficacy and utility preservation compared to state-of-the-art baselines. Our code is available at https://github.com/nxZhai/PALU.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 32 of its 52 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

32 of 52 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning30 Sep
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  3. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models5 Jun
  4. 2025Not All Tokens Are Meant to Be Forgotten3 Jun
  5. 2025Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning1 Jun
  6. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning17 May
  7. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  8. 2024On the Limitations and Prospects of Machine Unlearning for Generative AI1 Aug
  9. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  10. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  11. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  12. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
3 more
  1. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models8 Feb
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  3. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct

Other references · 17

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  7. 2023The Quantization Model of Neural Scalingcited by 138
  8. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  9. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 28
  10. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  11. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  12. 2025Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMscited by 18
5 more
  1. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  2. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
  3. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8
  4. 2024Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutionscited by 4
  5. 2025Debate over Mixed-knowledge: A Robust Multi-Agent Reasoning Framework for Incomplete Knowledge Graph Question Answeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
  2. 2026BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearningcited by 8
  3. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
  4. 2026Position: The Term "Machine Unlearning" Is Overused in LLMscited by 3
  5. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  6. 2026StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variabilitycited by 2
  7. 2026Mitigating Error Amplification in Fast Adversarial Trainingcited by 1
  8. 2026Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearningcited by 1
  9. 2026Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustnesscited by 0
  10. 2026TULIP: Targeted LLM Unlearning at Layers Identified Per-Inputcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution
  2. 2024A Closer Look at Machine Unlearning for Large Language Models
  3. 2026Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  5. 2026SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
  6. 2025Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
  7. 2026Representation-Guided Parameter-Efficient LLM Unlearning
  8. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges