Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks
Not on the map. Why?
Abstract (arXiv)
Experience replay plays a crucial role in improving the sample efficiency of deep reinforcement learning agents. Recent advances in experience replay propose using Mixup (Zhang et al., 2018) to further improve sample efficiency via synthetic sample generation. We build upon this technique with Neighborhood Mixup Experience Replay (NMER), a geometrically-grounded replay buffer that interpolates transitions with their closest neighbors in state-action space. NMER preserves a locally linear approximation of the transition manifold by only applying Mixup between transitions with vicinal state-action features. Under NMER, a given transition's set of state action neighbors is dynamic and episode agnostic, in turn encouraging greater policy generalizability via inter-episode interpolation. We combine our approach with recent off-policy deep reinforcement learning algorithms and evaluate on continuous control environments. We observe that NMER improves sample efficiency by an average 94% (TD3) and 29% (SAC) over baseline replay buffers, enabling agents to effectively recombine previous experiences and learn from limited data.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 18 May 2022.
Builds on
29 of 44 references.
References · 29
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016OpenAI Gymcited by 1,684
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2015Prioritized Experience Replaycited by 1,070
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
17 more
- 2020Reinforcement Learning with Augmented Datacited by 452
- 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
- 2019Challenges of Real-World Reinforcement Learningcited by 282
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
- 2016PlaNet - Photo Geolocation with Convolutional Neural Networkscited by 114
- 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
- 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
- 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
- 2019Data Efficient Reinforcement Learning for Legged Robotscited by 60
- 2019Experience Replay Optimizationcited by 47
- 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
- 2021Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Spacecited by 20
- 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14
- 2020Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUpcited by 5
- 2020Bootstrapping a DQN Replay Memory with Synthetic Experiencescited by 2
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.