mlsanityData to 14 Oct 2026

Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning

arXiv 2411.12155 ↗First posted 19 Nov 2024cs.LG cs.AI cs.ROcited by 10

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Predicting a sequence of actions has been crucial in the success of recent behavior cloning algorithms in robotics. Can similar ideas improve reinforcement learning (RL)? We answer affirmatively by observing that incorporating action sequences when predicting ground-truth return-to-go leads to lower validation loss. Motivated by this, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based RL algorithm that learns a critic network that outputs Q-values over a sequence of actions, i.e., explicitly training the value function to learn the consequence of executing action sequences. Our experiments show that CQN-AS outperforms several baselines on a variety of sparse-reward humanoid control and tabletop manipulation tasks from BiGym and RLBench.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 42 of its 58 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

42 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Continuous Control with Coarse-to-fine Reinforcement Learning10 Jul

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2016Layer Normalizationcited by 5,669
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2023Mastering Diverse Domains through World Modelscited by 1,253
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
29 more
  1. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  3. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  6. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  7. 2016The Option-Critic Architecturecited by 448
  8. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  13. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  14. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  15. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  16. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  17. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  18. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  19. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 114
  20. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  21. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 77
  22. 2023Imitation Bootstrapped Reinforcement Learningcited by 64
  23. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 44
  24. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 31
  25. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  26. 2024BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmarkcited by 30
  27. 2022Human-level Atari 200x fastercited by 28
  28. 2022Solving Continuous Control via Q-learningcited by 24
  29. 2023Reinforcement Learning with Simple Sequence Priorscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
  2. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimationcited by 4
  3. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 2
  4. 2026FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Controlcited by 1
  5. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learningcited by 1
  6. 2026PAC-ACT: Post-training Actor-Critic for Action Chunking Transformerscited by 1
  7. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  8. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  9. 2026Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvementcited by 0
  10. 2026Action Chunking Proximal Policy Optimization with Feedback Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
  2. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
  3. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
  4. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
  5. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
  6. 2026Accelerating Q-learning through Efficient Value-Sharing across Actions
  7. 2026Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning
  8. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning