Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 57 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
44 of 57 references.
Closest ancestors · 11
Same direction, newest first
- 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RL9 Oct
- 2025Reinforcement Learning with Action Chunking10 Jul
- 2025Horizon Reduction Makes RL Scalable4 Jun
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
- 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 33
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
- 2017Hindsight Experience Replaycited by 1,136
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
- 2016The Option-Critic Architecturecited by 448
21 more
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
- 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
- 2025Flow Q-Learningcited by 184
- 2020Revisiting Fundamentals of Experience Replaycited by 175
- 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
- 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
- 2021Hierarchical Skills for Efficient Explorationcited by 38
- 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
- 2025Decoupled Q-Chunkingcited by 20
- 2020Latent Skill Planning for Exploration and Transfercited by 19
- 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
- 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
- 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 17
- 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
- 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
- 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Chunk-Guided Q-Learning
- 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimation
- 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
- 2026Offline RL with Hierarchical Action Chunking
- 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization
- 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
- 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
- 2026Action Chunking Proximal Policy Optimization with Feedback Correction