mlsanityData to 7 Oct 2026

Continuous Control with Coarse-to-fine Reinforcement Learning

arXiv 2407.07787 ↗First posted 10 Jul 2024cs.RO cs.AI cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Despite recent advances in improving the sample-efficiency of reinforcement learning (RL) algorithms, designing an RL algorithm that can be practically deployed in real-world environments remains a challenge. In this paper, we present Coarse-to-fine Reinforcement Learning (CRL), a framework that trains RL agents to zoom-into a continuous action space in a coarse-to-fine manner, enabling the use of stable, sample-efficient value-based RL algorithms for fine-grained continuous control tasks. Our key idea is to train agents that output actions by iterating the procedure of (i) discretizing the continuous action space into multiple intervals and (ii) selecting the interval with the highest Q-value to further discretize at the next level. We then introduce a concrete, value-based algorithm within the CRL framework called Coarse-to-fine Q-Network (CQN). Our experiments demonstrate that CQN significantly outperforms RL and behavior cloning baselines on 20 sparsely-rewarded RLBench manipulation tasks with a modest number of environment interactions and expert demonstrations. We also show that CQN robustly learns to solve real-world manipulation tasks within a few minutes of online training.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 119 of its 141 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

119 of 141 references.

References · 119

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Layer Normalizationcited by 5,579
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  11. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  13. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  14. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  15. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  16. 2017Deep Reinforcement Learning that Matterscited by 727
  17. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  18. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  19. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  20. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  21. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  22. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  23. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  24. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  25. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  26. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  27. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  28. 2020Reinforcement Learning with Augmented Datacited by 452
  29. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  30. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  31. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  32. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  33. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  34. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  35. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  36. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  37. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  38. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 97
  2. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
  3. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  4. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  5. 2025Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Rewardcited by 13
  6. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  7. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  8. 2025FLAM: Foundation Model-Based Body Stabilization for Humanoid Locomotion and Manipulationcited by 2
  9. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  10. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1
  11. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learningcited by 1
  12. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
3 more
  1. 2025PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explorecited by 0
  2. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actionscited by 0
  3. 2026ZeroBot: Learning from Scratch in Minutes with Generative Real2Simcited by 0