Planning with Abstract Learned Models While Learning Transferable Subtasks
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. First posted on arXiv 16 Dec 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 2; 2023: 1.
4 papers.
- 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
- 2023Learning Rational Subgoals from Demonstrations and Instructionscited by 5
- 2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templatescited by 2
- 2022SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement Learningcited by 0
- 2023Learning Rational Subgoals from Demonstrations and Instructions9 Mar 2023
- 2022SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement Learning9 Mar 2022
- 2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates25 Feb 2022
- 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Options16 Jan 2021
Similar, not cited
Close in wording, no citation either way
- 2024Learning Abstract World Model for Value-preserving Planning with Options
- 2018Hierarchical Imitation and Reinforcement Learning
- 2026Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning
- 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
- 2019Model Primitive Hierarchical Lifelong Reinforcement Learning
- 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
- 2026In-Context Planning with Latent Temporal Abstractions
- 2022Hierarchical Decision Transformer