Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning
Not on the map. Why?
Abstract (arXiv)
In earthwork and construction, excavators often encounter large rocks mixed with various soil conditions, requiring skilled operators. This paper presents a framework for achieving autonomous excavation using reinforcement learning (RL) through a rock excavation simulator. In the simulation, resolution can be defined by the particle size/number in the whole soil space. Fine-resolution simulations closely mimic real-world behavior but demand significant calculation time and challenging sample collection, while coarse-resolution simulations enable faster sample collection but deviate from real-world behavior. To combine the advantages of both resolutions, we explore using policies developed in coarse-resolution simulations for pre-training in fine-resolution simulations. To this end, we propose a novel policy learning framework called Progressive-Resolution Policy Distillation (PRPD), which progressively transfers policies through some middle-resolution simulations with conservative policy transfer to avoid domain gaps that could lead to policy transfer failure. Validation in a rock excavation simulator and nine real-world rock environments demonstrated that PRPD reduced sampling time to less than 1/7 while maintaining task success rates comparable to those achieved through policy learning in a fine-resolution simulation. Additional videos and supplementary results are available on our project page: https://yuki-kadokawa.github.io/prpd/
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 66 references that are in the map. First posted on arXiv 10 Dec 2024.
Builds on
32 of 66 references.
References · 32
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
- 2019Model-Based Reinforcement Learning for Ataricited by 495
- 2017Robust Adversarial Reinforcement Learningcited by 439
- 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
- 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
20 more
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
- 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
- 2017Teacher-Student Curriculum Learningcited by 197
- 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
- 2019Distilling Policy Distillationcited by 81
- 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
- 2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
- 2020Dual Policy Distillationcited by 25
- 2023Ensemble Reinforcement Learning: A Surveycited by 16
- 2019Deep Conservative Policy Iterationcited by 14
- 2023Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulationcited by 9
- 2022Disturbance-Injected Robust Imitation Learning with Task Achievementcited by 8
- 2022Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomizationcited by 7
- 2022Excavation Reinforcement Learning Using Geometric Representationcited by 5
- 2021Ensemble and Auxiliary Tasks for Data-Efficient Deep Reinforcement Learningcited by 2
- 2021Real-Time Motion Planning of a Hydraulic Excavator using Trajectory Optimization and Model Predictive Controlcited by 2
- 2021Cautious Actor-Criticcited by 2
- 2023Granular Gym: High Performance Simulation for Robotic Tasks with Granular Materialscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.