mlsanityData to 14 Oct 2026

Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks

arXiv 2504.11247 ↗First posted 15 Apr 2025cs.RO cs.CV cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Hindsight Experience Replay (HER) is widely regarded as the state-of-the-art algorithm for achieving sample-efficient multi-goal reinforcement learning (RL) in robotic manipulation tasks with binary rewards. HER facilitates learning from failed attempts by replaying trajectories with redefined goals. However, it relies on a heuristic-based replay method that lacks a principled framework. To address this limitation, we introduce a novel replay strategy, "Next-Future", which focuses on rewarding single-step transitions. This approach significantly enhances sample efficiency and accuracy in learning multi-goal Markov decision processes (MDPs), particularly under stringent accuracy requirements -- a critical aspect for performing complex and precise robotic-arm tasks. We demonstrate the efficacy of our method by highlighting how single-step learning enables improved value approximation within the multi-goal RL framework. The performance of the proposed replay strategy is evaluated across eight challenging robotic manipulation tasks, using ten random seeds for training. Our results indicate substantial improvements in sample efficiency for seven out of eight tasks and higher success rates in six tasks. Furthermore, real-world experiments validate the practical feasibility of the learned policies, demonstrating the potential of "Next-Future" in solving complex robotic-arm tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 31 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

23 of 31 references.

References · 23

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  10. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  11. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  12. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
11 more
  1. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  2. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  3. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 59
  4. 2023Neural Implicit Dense Semantic SLAMcited by 26
  5. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  6. 2018Energy-Based Hindsight Experience Prioritizationcited by 21
  7. 2024ICGNet: A Unified Approach for Instance-Centric Graspingcited by 16
  8. 2022Organic Priors in Non-Rigid Structure from Motioncited by 15
  9. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challengescited by 13
  10. 2021A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenescited by 12
  11. 2020Dense Non-Rigid Structure from Motion: A Manifold Viewpointcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation
  2. 2019Goal-conditioned Imitation Learning
  3. 2025CONTHER: Context-Aware Reinforcement Learning for Robotic Manipulation with Sparse Rewards
  4. 2026E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation
  5. 2025Goal-based Self-Adaptive Generative Adversarial Imitation Learning (Goal-SAGAIL) for Multi-goal Robotic Manipulation Tasks
  6. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
  7. 2025Robotic Long-Horizon Manipulation with Bayesian Non-parametric Skill Priors
  8. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning