# Exact Unlearning in Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.04182 · HTML: https://mlsanity.com/paper/2606.04182/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)
- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 19 of 19)

- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [Machine Unlearning via Algorithmic Stability](https://mlsanity.com/paper/2102.13179/) (2021)
- [Finite Sample Differentially Private Confidence Intervals](https://mlsanity.com/paper/1711.03908/) (2017)
- [Differentially Private Contextual Linear Bandits](https://mlsanity.com/paper/1810.00068/) (2018)
- [Private Reinforcement Learning with PAC and Regret Guarantees](https://mlsanity.com/paper/2009.09052/) (2020)
- [Differentially Private Regret Minimization in Episodic Markov Decision Processes](https://mlsanity.com/paper/2112.10599/) (2021)
- [Differentially Private Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2201.07052/) (2022)
- [The Utility and Complexity of in- and out-of-Distribution Machine Unlearning](https://mlsanity.com/paper/2412.09119/) (2024)
- [Reinforcement Unlearning](https://mlsanity.com/paper/2312.15910/) (2023)
- [Near-Optimal Differentially Private Reinforcement Learning](https://mlsanity.com/paper/2212.04680/) (2022)
- [From Adaptive Query Release to Machine Unlearning](https://mlsanity.com/paper/2307.11228/) (2023)
- [When to Forget? Complexity Trade-offs in Machine Unlearning](https://mlsanity.com/paper/2502.17323/) (2025)
- [Forgettable Federated Linear Learning with Certified Data Unlearning](https://mlsanity.com/paper/2306.02216/) (2023)

## Built on by (most cited first, 1 of 1)

- [ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control](https://mlsanity.com/paper/2609.28378/) (2026)

## Abstract (arXiv)

We formulate the problem of exact unlearning in reinforcement learning, where the goal is to design an efficient framework that enables the removal of any user's data upon deletion request, i.e., the online learner's output after unlearning is indistinguishable from what would have been produced had the deleted user never interacted with the learner. For any ρ >0, we show that there exists a reinforcement learning (RL) algorithm that is ρ-TV-stable and supports an exact unlearning procedure whose expected computational cost is only a ρ √ln T fraction of the computational cost of retraining from scratch. We construct such a ρ-TV-stable RL algorithm for tabular Markov decision processes (MDPs), which achieves a regret bound of O(H2 √SAT + H3 S2 A + H2.5 S2 A/ρ), where S, A, H, and T denote the number of states, the number of actions, the episode horizon, and the number of episodes, respectively. We also establish a lower bound of Omega(H√!SAT! +! SAH/ρ) for ρ-TV-stable RL algorithms, showing that our algorithm is nearly minimax optimal.