# ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search

arXiv: https://arxiv.org/abs/1811.02696 · HTML: https://mlsanity.com/paper/1811.02696/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learning](https://mlsanity.com/paper/2201.13357/) (2022)
- [Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2208.12328/) (2022)
- [MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.10552/) (2021)
- [Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control](https://mlsanity.com/paper/2207.13730/) (2022)
- [Wasserstein Barycenter Soft Actor-Critic](https://mlsanity.com/paper/2506.10167/) (2025)
- [Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies](https://mlsanity.com/paper/2508.03194/) (2025)
- [Retry Policy Gradients in Continuous Action Spaces](https://mlsanity.com/paper/2606.05888/) (2026)
- [TASAC: a twin-actor reinforcement learning framework with stochastic policy for batch process control](https://mlsanity.com/paper/2204.10685/) (2022)
- [One is More: Diverse Perspectives within a Single Network for Efficient DRL](https://mlsanity.com/paper/2310.14009/) (2023)
- [OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Search](https://mlsanity.com/paper/2609.40035/) (2026)

## Abstract (arXiv)

In this paper, we propose an actor ensemble algorithm, named ACE, for continuous control with a deterministic policy in reinforcement learning. In ACE, we use actor ensemble (i.e., multiple actors) to search the global maxima of the critic. Besides the ensemble perspective, we also formulate ACE in the option framework by extending the option-critic architecture with deterministic intra-option policies, revealing a relationship between ensemble and options. Furthermore, we perform a look-ahead tree search with those actors and a learned value prediction model, resulting in a refined value estimation. We demonstrate a significant performance boost of ACE over DDPG and its variants in challenging physical robot simulators.