# Shaped Policy Search for Evolutionary Strategies using Waypoints

arXiv: https://arxiv.org/abs/2105.14639 · HTML: https://mlsanity.com/paper/2105.14639/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 30 May 2021.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Reinforcement Learning from Imperfect Demonstrations](https://mlsanity.com/paper/1802.05313/) (2018)
- [RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration](https://mlsanity.com/paper/1906.07372/) (2019)
- [Hybrid Reinforcement Learning with Expert State Sequences](https://mlsanity.com/paper/1903.04110/) (2019)
- [robo-gym -- An Open Source Toolkit for Distributed Deep Reinforcement Learning on Real and Simulated Robots](https://mlsanity.com/paper/2007.02753/) (2020)
- [Attentive Feature Reuse for Multi Task Meta learning](https://mlsanity.com/paper/2006.07438/) (2020)
- [Sparse Allreduce: Efficient Scalable Communication for Power-Law Data](https://mlsanity.com/paper/1312.3020/) (2013)
- [Meta Adaptation using Importance Weighted Demonstrations](https://mlsanity.com/paper/1911.10322/) (2019)

## Built on by (most cited first, 1 of 1)

- [Bird's Eye View Based Pretrained World model for Visual Navigation](https://mlsanity.com/paper/2310.18847/) (2023)

## Abstract (arXiv)

In this paper, we try to improve exploration in Blackbox methods, particularly Evolution strategies (ES), when applied to Reinforcement Learning (RL) problems where intermediate waypoints/subgoals are available. Since Evolutionary strategies are highly parallelizable, instead of extracting just a scalar cumulative reward, we use the state-action pairs from the trajectories obtained during rollouts/evaluations, to learn the dynamics of the agent. The learnt dynamics are then used in the optimization procedure to speed-up training. Lastly, we show how our proposed approach is universally applicable by presenting results from experiments conducted on Carla driving and UR5 robotic arm simulators.