mlsanityData to 14 Oct 2026

Parallel Heuristic Search as Inference for Actor-Critic Reinforcement Learning Models

arXiv 2509.25402 ↗First posted 29 Sep 2025cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Actor-Critic models are a class of model-free deep reinforcement learning (RL) algorithms that have demonstrated effectiveness across various robot learning tasks. While considerable research has focused on improving training stability and data sampling efficiency, most deployment strategies have remained relatively simplistic, typically relying on direct actor policy rollouts. In contrast, we propose pachs (textitParallel textitActor-textitCritic textitHeuristic textitSearch), an efficient parallel best-first search algorithm for inference that leverages both components of the actor-critic architecture: the actor network generates actions, while the critic network provides cost-to-go estimates to guide the search. Two levels of parallelism are employed within the search -- actions and cost-to-go estimates are generated in batches by the actor and critic networks respectively, and graph expansion is distributed across multiple threads. We demonstrate the effectiveness of our approach in robotic manipulation tasks, including collision-free motion planning and contact-rich interactions such as non-prehensile pushing. Visit p-achs.github.io for demonstrations and examples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 30 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

14 of 30 references.

References · 14

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  5. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  7. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 59
  8. 2023Motions in Microseconds via Vectorized Sampling-Based Planningcited by 17
  9. 2022ePA*SE: Edge-based Parallel A* for Slow Evaluationscited by 4
  10. 2023Learning Local Heuristics for Search-Based Navigation Planningcited by 3
  11. 2022Parallel Monte Carlo Tree Search with Batched Rigid-body Simulations for Speeding up Long-Horizon Episodic Robot Planningcited by 2
  12. 2023GePA*SE: Generalized Edge-Based Parallel A* for Slow Evaluationscited by 2
2 more
  1. 2023A-ePA*SE: Anytime Edge-Based Parallel A* for Slow Evaluationscited by 1
  2. 2024SLOPE: Search with Learned Optimal Pruning-based Expansioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  2. 2026Actor-Critic Pretraining for Proximal Policy Optimization
  3. 2023Language-Conditioned Semantic Search-Based Policy for Robotic Manipulation Tasks
  4. 2025Comparative Analysis of Parameterized Action Actor-Critic Reinforcement Learning Algorithms for Web Search Match Plan Generation
  5. 2021Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
  6. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies
  7. 2025Actor-Critic without Actor
  8. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning