mlsanityData to 7 Oct 2026

Chunk-Guided Q-Learning

arXiv 2603.13971 ↗First posted 14 Mar 2026cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 43 references that are in the map. First posted on arXiv 14 Mar 2026.

Builds on

39 of 43 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Scalable Offline Model-Based RL with Action Chunks8 Dec
  2. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RL9 Oct
  3. 2025Reinforcement Learning with Action Chunking10 Jul
  4. 2025Horizon Reduction Makes RL Scalable4 Jun
  5. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  6. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  7. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  8. 2022Efficient Offline Policy Optimization with a Learned Model12 Oct
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021Offline RL Without Off-Policy Evaluation16 Jun
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
4 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2016Layer Normalizationcited by 5,579
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
11 more
  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  3. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  4. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  5. 2025Flow Q-Learningcited by 162
  6. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  7. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  8. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  9. 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
  10. 2025Decoupled Q-Chunkingcited by 18
  11. 2025Transitive RL: Value Learning via Divide and Conquercited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  2. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learningcited by 0
  3. 2026Action Chunking Proximal Policy Optimization with Feedback Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
  2. 2026Offline RL with Hierarchical Action Chunking
  3. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning
  5. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimation
  6. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  8. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking