# Decoupled Q-Chunking

arXiv: https://arxiv.org/abs/2512.10926 · HTML: https://mlsanity.com/paper/2512.10926/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2025.

## Built on by (most cited first, 20 of 20)

- [Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning](https://mlsanity.com/paper/2606.11087/) (2026)
- [Chunk-Guided Q-Learning](https://mlsanity.com/paper/2603.13971/) (2026)
- [Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference](https://mlsanity.com/paper/2605.26552/) (2026)
- [Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL](https://mlsanity.com/paper/2608.30640/) (2026)
- [Refining Compositional Diffusion for Reliable Long-Horizon Planning](https://mlsanity.com/paper/2605.03075/) (2026)
- [Offline Reinforcement Learning with Universal Horizon Models](https://mlsanity.com/paper/2605.15603/) (2026)
- [Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency](https://mlsanity.com/paper/2608.23831/) (2026)
- [Action-Sufficient Goal Representations](https://mlsanity.com/paper/2601.22496/) (2026)
- [Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2605.05544/) (2026)
- [Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradients](https://mlsanity.com/paper/2605.10909/) (2026)
- [Offline RL with Hierarchical Action Chunking](https://mlsanity.com/paper/2607.20834/) (2026)
- [SmoothRL: Online Reinforcement Learning During Asynchronous Execution](https://mlsanity.com/paper/2608.29768/) (2026)
- [ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network](https://mlsanity.com/paper/2605.11009/) (2026)
- [Implicit Action Chunking for Smooth Continuous Control](https://mlsanity.com/paper/2605.19592/) (2026)
- [SegBench-GC: Testing Segmentation Invariance in Multi-Step Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2608.27678/) (2026)
- [Lifted Bellman Linear Programming for Offline Reinforcement Learning](https://mlsanity.com/paper/2609.24489/) (2026)
- [GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences](https://mlsanity.com/paper/2609.33259/) (2026)
- [Action Chunking Proximal Policy Optimization with Feedback Correction](https://mlsanity.com/paper/2609.36250/) (2026)
- [Temporally Interpretable Differentiable Decision Trees](https://mlsanity.com/paper/2610.10367/) (2026)
- [Chunk-wise Model-based Policy Optimization](https://mlsanity.com/paper/2610.14564/) (2026)

## Abstract (arXiv)

Temporal-difference (TD) methods learn state and action values efficiently by bootstrapping from their own future value predictions, but such a self-bootstrapping mechanism is prone to bootstrapping bias, where the errors in the value targets accumulate across steps and result in biased value estimates. Recent work has proposed to use chunked critics, which estimate the value of short action sequences ("chunks") rather than individual actions, speeding up value backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which can be sub-optimal for environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when a partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks. We evaluate our method on challenging, long-horizon offline goal-conditioned tasks and show that it reliably outperforms prior methods. Code: github.com/ColinQiyangLi/dqc.