Data to 7 Oct 2026

Reinforcement Learning with Action Chunking

arXiv 2507.07969 ↗First posted 10 Jul 2025cs.LG cs.AI cs.ROcited by 97

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a 'chunked' action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased n-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 64 of its 92 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

64 of 92 references.

Closest ancestors · 15

Same direction, newest first

3 more

Other references · 49

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
2015Continuous control with deep reinforcement learningcited by 3,279
2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
2020Learning to summarize from human feedbackcited by 2,562
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2018Data-Efficient Hierarchical Reinforcement Learningcited by 436

37 more

2016The Option-Critic Architecturecited by 433
2023Efficient Online Reinforcement Learning with Offline Datacited by 332
2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
2024Diffusion Policy Policy Optimizationcited by 255
2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2020Revisiting Fundamentals of Experience Replaycited by 171
2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
2025Flow Q-Learningcited by 162
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
2017Value Prediction Networkcited by 117
2018Neural probabilistic motor primitives for humanoid controlcited by 96
2024Foundation Policies with Hilbert Representationscited by 89
2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
2020Learning Robot Skills with Temporal Variational Inferencecited by 54
2017Multi-Level Discovery of Deep Optionscited by 50
2019Variational Temporal Abstractioncited by 37
2021Hierarchical Skills for Efficient Explorationcited by 37
2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
2023Accelerating Exploration with Unlabeled Prior Datacited by 21
2020Latent Skill Planning for Exploration and Transfercited by 19
2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
2023One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformerscited by 17
2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
2016Deep Reinforcement Learning With Macro-Actionscited by 13
2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 10
2016Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clusteringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 77.

97 papers.

Sort:

By area:

2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
2026Q-learning with Adjoint Matchingcited by 33
2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 13
2025Value Flowscited by 12
2025Scalable Offline Model-Based RL with Action Chunkscited by 12
2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
2025Multi-agent Coordination via Flow Matchingcited by 9

38 more

2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
2026Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive sciencecited by 8
2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
2026RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Frameworkcited by 7
2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 6
2026StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observationcited by 6
2025SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Explorationcited by 5
2025Human-in-the-loop Online Rejection Sampling for Robotic Manipulationcited by 5
2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
2025GR-Dexter Technical Reportcited by 4
2026Reparameterization Flow Policy Optimizationcited by 4
2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
2026Chunk-Guided Q-Learningcited by 3
2026Trust Region Q Adjoint Matchingcited by 3
2025Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigationcited by 2
2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
2026Learning Visual Feature-Based World Models via Residual Latent Actioncited by 2
2026DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimizationcited by 2
2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
2026Reversal Q-Learningcited by 2
2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulationcited by 2
2026Action-Sufficient Goal Representationscited by 1
2026Adapting Reinforcement Learning for Path Planning in Constrained Parking Scenarioscited by 1
2026Latent Policy Steering through One-Step Flow Policiescited by 1
2026Tube Diffusion Policy: Reactive Visual-Tactile Policy Learning for Contact-rich Manipulationcited by 1

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way