mlsanityData to 14 Oct 2026

CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality

arXiv 2211.06625 ↗First posted 12 Nov 2022cs.RO cs.LG math.OCcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel algorithm for the continuous control of dynamical systems that combines Trajectory Optimization (TO) and Reinforcement Learning (RL) in a single framework. The motivations behind this algorithm are the two main limitations of TO and RL when applied to continuous nonlinear systems to minimize a non-convex cost function. Specifically, TO can get stuck in poor local minima when the search is not initialized close to a "good" minimum. On the other hand, when dealing with continuous state and control spaces, the RL training process may be excessively long and strongly dependent on the exploration strategy. Thus, our algorithm learns a "good" control policy via TO-guided RL policy search that, when used as initial guess provider for TO, makes the trajectory optimization process less prone to converge to poor local optima. Our method is validated on several reaching problems featuring non-convex obstacle avoidance with different dynamical systems, including a car model with 6D state, and a 3-joint planar manipulator. Our results show the great capabilities of CACTO in escaping local minima, while being more computationally efficient than the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) RL algorithms.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

21 of 42 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  8. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  9. 2017Sobolev Training for Neural Networkscited by 143
  10. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  11. 2018Collapse of Deep and Narrow Neural Netscited by 37
  12. 2019Crocoddyl: An Efficient and Versatile Framework for Multi-Contact Optimal Controlcited by 34
9 more
  1. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  2. 2017Whole-Body Nonlinear Model Predictive Control Through Contacts for Quadrupedscited by 20
  3. 2019The problem with DDPG: understanding failures in deterministic environments with sparse rewardscited by 20
  4. 2019Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environmentscited by 14
  5. 2021Ergodic Exploration using Tensor Train: Applications in Insertion Taskscited by 7
  6. 2019MPC for Humanoid Gait Generation: Stability and Feasibilitycited by 6
  7. 2020A Feasibility-Driven Approach to Control-Limited DDPcited by 6
  8. 2020Interior Point Differential Dynamic Programmingcited by 4
  9. 2021Combined Sampling and Optimization Based Planning for Legged-Wheeled Robotscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 5.

12 papers.

  1. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 26
  2. 2024CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Controlcited by 6
  3. 2023CACTO-SL: Using Sobolev Learning to improve Continuous Actor-Critic with Trajectory Optimizationcited by 4
  4. 2025Guided Reinforcement Learning for Omnidirectional 3D Jumping in Quadruped Robotscited by 4
  5. 2026Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulationcited by 3
  6. 2026CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimizationcited by 2
  7. 2023Efficient Reinforcement Learning for Jumping Monopodscited by 1
  8. 2026Accelerating trajectory optimization with Sobolev-trained diffusion policiescited by 1
  9. 2026CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learningcited by 1
  10. 2024Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learningcited by 0
  11. 2025Re4MPC: Reactive Nonlinear MPC for Multi-model Motion Planning via Deep Reinforcement Learningcited by 0
  12. 2026RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environmentscited by 0