# Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks with Base Controllers

arXiv: https://arxiv.org/abs/2011.12105 · HTML: https://mlsanity.com/paper/2011.12105/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control](https://mlsanity.com/paper/2210.01525/) (2022)
- [ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay](https://mlsanity.com/paper/2511.19033/) (2025)
- [Auxiliary Task-based Deep Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2302.14312/) (2023)
- [ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control](https://mlsanity.com/paper/2410.05225/) (2024)
- [Automating Deformable Gasket Assembly](https://mlsanity.com/paper/2408.12593/) (2024)
- [High-Fidelity One-Step Generative Visuomotor Policy via Recursive Correction, Frequency Consistency, and Contrastive Flow Matching](https://mlsanity.com/paper/2607.03865/) (2026)
- [Learning from demonstrations: An intuitive VR environment for imitation learning of construction robots](https://mlsanity.com/paper/2305.14584/) (2023)
- [D3Grasp: Diverse and Deformable Dexterous Grasping for General Objects](https://mlsanity.com/paper/2509.19892/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) enables robots to perform some intelligent tasks end-to-end. However, there are still many challenges for long-horizon sparse-reward robotic manipulator tasks. On the one hand, a sparse-reward setting causes exploration inefficient. On the other hand, exploration using physical robots is of high cost and unsafe. In this paper, we propose a method of learning long-horizon sparse-reward tasks utilizing one or more existing traditional controllers named base controllers in this paper. Built upon Deep Deterministic Policy Gradients (DDPG), our algorithm incorporates the existing base controllers into stages of exploration, value learning, and policy update. Furthermore, we present a straightforward way of synthesizing different base controllers to integrate their strengths. Through experiments ranging from stacking blocks to cups, it is demonstrated that the learned state-based or image-based policies steadily outperform base controllers. Compared to previous works of learning from demonstrations, our method improves sample efficiency by orders of magnitude and improves the performance. Overall, our method bears the potential of leveraging existing industrial robot manipulation systems to build more flexible and intelligent controllers.