# Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.01437 · HTML: https://mlsanity.com/paper/2503.01437/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 37)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity](https://mlsanity.com/paper/1902.05605/) (2019)
- [Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP](https://mlsanity.com/paper/1906.02768/) (2019)
- [Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers](https://mlsanity.com/paper/2005.06870/) (2020)
- [The Difficulty of Training Sparse Neural Networks](https://mlsanity.com/paper/1906.10732/) (2019)
- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [The Difficulty of Passive Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2110.14020/) (2021)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Training Larger Networks for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.07920/) (2021)
- [Sample-Efficient Automated Deep Reinforcement Learning](https://mlsanity.com/paper/2009.01555/) (2020)
- [PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning](https://mlsanity.com/paper/2001.05012/) (2020)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)

## Built on by (most cited first, 6 of 6)

- [FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control](https://mlsanity.com/paper/2604.04539/) (2026)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)
- [Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning](https://mlsanity.com/paper/2510.02590/) (2025)
- [Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies](https://mlsanity.com/paper/2512.04279/) (2025)
- [Gradient Iterated Temporal-Difference Learning](https://mlsanity.com/paper/2603.07833/) (2026)
- [Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization](https://mlsanity.com/paper/2606.26002/) (2026)

## Abstract (arXiv)

Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De Q-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari 2600 benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.