# Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward

arXiv: https://arxiv.org/abs/2508.11143 · HTML: https://mlsanity.com/paper/2508.11143/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation](https://mlsanity.com/paper/2408.11812/) (2024)
- [BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark](https://mlsanity.com/paper/2407.07788/) (2024)
- [InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation](https://mlsanity.com/paper/2409.07914/) (2024)
- [Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators](https://mlsanity.com/paper/2305.03270/) (2023)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [Reinforcement Learning with Simple Sequence Priors](https://mlsanity.com/paper/2305.17109/) (2023)
- [Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers](https://mlsanity.com/paper/2503.13916/) (2025)

## Built on by (most cited first, 13 of 13)

- [STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Models](https://mlsanity.com/paper/2512.05107/) (2025)
- [RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation](https://mlsanity.com/paper/2603.11106/) (2026)
- [SEAR: Sample Efficient Action Chunking Reinforcement Learning](https://mlsanity.com/paper/2603.01891/) (2026)
- [Adaptive Action Chunking via Multi-Chunk Q Value Estimation](https://mlsanity.com/paper/2605.10044/) (2026)
- [Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution](https://mlsanity.com/paper/2608.03483/) (2026)
- [PAC-ACT: Post-training Actor-Critic for Action Chunking Transformers](https://mlsanity.com/paper/2607.09590/) (2026)
- [StructRL: Structured Action-Space Exploration for Flow-Based VLAs](https://mlsanity.com/paper/2608.15139/) (2026)
- [ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network](https://mlsanity.com/paper/2605.11009/) (2026)
- [Implicit Action Chunking for Smooth Continuous Control](https://mlsanity.com/paper/2605.19592/) (2026)
- [Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning](https://mlsanity.com/paper/2607.23726/) (2026)
- [Harnessing human expertise for high-precision robotic assembly in industrialized construction: A sample-efficient installer-in-the-loop interactive reinforcement learning framework](https://mlsanity.com/paper/2609.13234/) (2026)
- [Learning from Mixed-Quality Deployment Experience for Robot Manipulation](https://mlsanity.com/paper/2609.29000/) (2026)
- [Action Chunking Proximal Policy Optimization with Feedback Correction](https://mlsanity.com/paper/2609.36250/) (2026)

## Abstract (arXiv)

Existing reinforcement learning (RL) methods struggle with long-horizon robotic manipulation tasks, particularly those involving sparse rewards. While action chunking is a promising paradigm for robotic manipulation, using RL to directly learn continuous action chunks in a stable and data-efficient manner remains a critical challenge. This paper introduces AC3 (Actor-Critic for Continuous Chunks), a novel RL framework that learns to generate high-dimensional, continuous action sequences. To make this learning process stable and data-efficient, AC3 incorporates targeted stabilization mechanisms for both the actor and the critic. First, to ensure reliable policy improvement, the actor is trained with an asymmetric update rule, learning exclusively from successful trajectories. Second, to enable effective value learning despite sparse rewards, the critic's update is stabilized using intra-chunk n-step returns and further enriched by a self-supervised module providing intrinsic rewards at anchor points aligned with each action chunk. We conducted extensive experiments on 25 tasks from the BiGym and RLBench benchmarks. Results show that by using only a few demonstrations and a simple model architecture, AC3 achieves superior success rates on most tasks, validating its effective design.