# Model Learning for Look-ahead Exploration in Continuous Control

arXiv: https://arxiv.org/abs/1811.08086 · HTML: https://mlsanity.com/paper/1811.08086/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2018.

## Built on by (most cited first, 4 of 4)

- [Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2404.01943/) (2024)
- [Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning](https://mlsanity.com/paper/2201.09765/) (2022)
- [Learning Extrinsic Dexterity with Parameterized Manipulation Primitives](https://mlsanity.com/paper/2310.17785/) (2023)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)

## Abstract (arXiv)

We propose an exploration method that incorporates look-ahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies . Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics, i.e., the state transition caused by a (complete) skill execution, are learnt and are unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions, and thus the resulting policies does not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parametrized skills as building blocks of the policy itself, as opposed to guiding exploration. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parameterized skills as building blocks of the policy itself, as opposed to guiding exploration.