mlsanityData to 14 Oct 2026

D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation

arXiv 2603.27346 ↗First posted 28 Mar 2026cs.RO cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 17 references that are in the map. First posted on arXiv 28 Mar 2026.

Builds on

12 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes7 Aug

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2015Prioritized Experience Replaycited by 1,099
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  9. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  10. 2022Actor Prioritized Experience Replaycited by 20
  11. 2024Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policiescited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks
  2. 2026E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation
  3. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation
  4. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation
  5. 2026Demonstration-Calibrated Port-Hamiltonian Retuning for Manipulation Policies
  6. 2025CONTHER: Context-Aware Reinforcement Learning for Robotic Manipulation with Sparse Rewards
  7. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
  8. 2026RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation