# CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization

arXiv: https://arxiv.org/abs/2602.19699 · HTML: https://mlsanity.com/paper/2602.19699/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Model Predictive Path Integral Control using Covariance Variable Importance Sampling](https://mlsanity.com/paper/1509.01149/) (2015)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Optimization-Based Control for Dynamic Legged Robots](https://mlsanity.com/paper/2211.11644/) (2022)
- [MPC-Net: A First Principles Guided Policy Search](https://mlsanity.com/paper/1909.05197/) (2019)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning](https://mlsanity.com/paper/2403.18765/) (2024)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality](https://mlsanity.com/paper/2211.06625/) (2022)
- [Sample Efficient Deep Reinforcement Learning via Local Planning](https://mlsanity.com/paper/2301.12579/) (2023)
- [Contrastive Initial State Buffer for Reinforcement Learning](https://mlsanity.com/paper/2309.09752/) (2023)
- [A Safe Reinforcement Learning driven Weights-varying Model Predictive Control for Autonomous Vehicle Motion Control](https://mlsanity.com/paper/2402.02624/) (2024)
- [AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control](https://mlsanity.com/paper/2406.03995/) (2024)
- [Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control](https://mlsanity.com/paper/2209.09006/) (2022)
- [Imitation Learning from Nonlinear MPC via the Exact Q-Loss and its Gauss-Newton Approximation](https://mlsanity.com/paper/2304.01782/) (2023)
- [CACTO-SL: Using Sobolev Learning to improve Continuous Actor-Critic with Trajectory Optimization](https://mlsanity.com/paper/2312.10666/) (2023)
- [Infinite-Horizon Value Function Approximation for Model Predictive Control](https://mlsanity.com/paper/2502.06760/) (2025)
- [Improved Exploring Starts by Kernel Density Estimation-Based State-Space Coverage Acceleration in Reinforcement Learning](https://mlsanity.com/paper/2105.08990/) (2021)
- [Where2Start: Leveraging initial States for Robust and Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2311.15089/) (2023)

## Built on by (most cited first, 2 of 2)

- [TurboMPC: Fast, Scalable, and Differentiable Model Predictive Control on the GPU](https://mlsanity.com/paper/2606.24039/) (2026)
- [CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2609.08853/) (2026)

## Abstract (arXiv)

Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.