# HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym

arXiv: https://arxiv.org/abs/2505.22597 · HTML: https://mlsanity.com/paper/2505.22597/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [PDDLGym: Gym Environments from PDDL Problems](https://mlsanity.com/paper/2002.06432/) (2020)
- [Discovering User-Interpretable Capabilities of Black-Box Planning Agents](https://mlsanity.com/paper/2107.13668/) (2021)
- [Learning for Multi-robot Cooperation in Partially Observable Stochastic Environments with Macro-actions](https://mlsanity.com/paper/1707.07399/) (2017)
- [pyRDDLGym: From RDDL to Gym Environments](https://mlsanity.com/paper/2211.05939/) (2022)
- [NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds](https://mlsanity.com/paper/2401.03546/) (2024)
- [An Accurate HDDL Domain Learning Algorithm from Partial and Noisy Observations](https://mlsanity.com/paper/2206.06882/) (2022)

## Built on by (most cited first, 1 of 1)

- [Autonomous Assessment of Generalizability of AI Agent Capabilities](https://mlsanity.com/paper/2512.16733/) (2025)

## Abstract (arXiv)

In recent years, reinforcement learning (RL) methods have been widely tested using tools like OpenAI Gym, though many tasks in these environments could also benefit from hierarchical planning. However, there is a lack of a tool that enables seamless integration of hierarchical planning with RL. Hierarchical Domain Definition Language (HDDL), used in classical planning, introduces a structured approach well-suited for model-based RL to address this gap. To bridge this integration, we introduce HDDLGym, a Python-based tool that automatically generates OpenAI Gym environments from HDDL domains and problems. HDDLGym serves as a link between RL and hierarchical planning, supporting multi-agent scenarios and enabling collaborative planning among agents. This paper provides an overview of HDDLGym's design and implementation, highlighting the challenges and design choices involved in integrating HDDL with the Gym interface, and applying RL policies to support hierarchical planning. We also provide detailed instructions and demonstrations for using the HDDLGym framework, including how to work with existing HDDL domains and problems from International Planning Competitions, exemplified by the Transport domain. Additionally, we offer guidance on creating new HDDL domains for multi-agent scenarios and demonstrate the practical use of HDDLGym in the Overcooked domain. By leveraging the advantages of HDDL and Gym, HDDLGym aims to be a valuable tool for studying RL in hierarchical planning, particularly in multi-agent contexts.