# Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making

arXiv: https://arxiv.org/abs/2312.11027 · HTML: https://mlsanity.com/paper/2312.11027/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 65 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 41)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [World Model as a Graph: Learning Latent Landmarks for Planning](https://mlsanity.com/paper/2011.12491/) (2020)
- [Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors](https://mlsanity.com/paper/2006.13205/) (2020)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023)
- [Learning Actionable Representations with Goal-Conditioned Policies](https://mlsanity.com/paper/1811.07819/) (2018)
- [Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies](https://mlsanity.com/paper/1807.07665/) (2018)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2002.12361/) (2020)
- [Evaluating the Robustness of Collaborative Agents](https://mlsanity.com/paper/2101.05507/) (2021)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation](https://mlsanity.com/paper/2109.08771/) (2021)
- [Transferable Task Execution from Pixels through Deep Planning Domain Learning](https://mlsanity.com/paper/2003.03726/) (2020)
- [Reward-Respecting Subtasks for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2202.03466/) (2022)
- [Model-Based Offline Meta-Reinforcement Learning with Regularization](https://mlsanity.com/paper/2202.02929/) (2022)
- [LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.02561/) (2022)

## Abstract (arXiv)

Many complicated real-world tasks can be broken down into smaller, more manageable parts, and planning with prior knowledge extracted from these simplified pieces is crucial for humans to make accurate decisions. However, replicating this process remains a challenge for AI agents and naturally raises two questions: How to extract discriminative knowledge representation from priors? How to develop a rational plan to decompose complex problems? Most existing representation learning methods employing a single encoder structure are fragile and sensitive to complex and diverse dynamics. To address this issue, we introduce a multiple-encoder and individual-predictor regime to learn task-essential representations from sufficient data for simple subtasks. Multiple encoders can extract adequate task-relevant dynamics without confusion, and the shared predictor can discriminate the task characteristics. We also use the attention mechanism to generate a top-k subtask planning tree, which customizes subtask execution plans in guiding complex decisions on unseen tasks. This process enables forward-looking and globality by flexibly adjusting the depth and width of the planning tree. Empirical results on a challenging platform composed of some basic simple tasks and combinatorially rich synthetic tasks consistently outperform some competitive baselines and demonstrate the benefits of our design.