# Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning

arXiv: https://arxiv.org/abs/2311.09830 · HTML: https://mlsanity.com/paper/2311.09830/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 14 of 14)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://mlsanity.com/paper/2009.00751/) (2020)

## Built on by (most cited first, 24 of 24)

- [ACPBench: Reasoning about Action, Change, and Planning](https://mlsanity.com/paper/2410.05669/) (2024)
- [Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming](https://mlsanity.com/paper/2410.12112/) (2024)
- [Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanism](https://mlsanity.com/paper/2511.17198/) (2025)
- [PROC2PDDL: Open-Domain Planning Representations from Texts](https://mlsanity.com/paper/2403.00092/) (2024)
- [PDDLEGO: Iterative Planning in Textual Environments](https://mlsanity.com/paper/2405.19793/) (2024)
- [Seemingly Simple Planning Problems are Computationally Challenging: The Countdown Game](https://mlsanity.com/paper/2508.02900/) (2025)
- [GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps](https://mlsanity.com/paper/2410.07765/) (2024)
- [Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study](https://mlsanity.com/paper/2507.23589/) (2025)
- [Improved Generalized Planning with LLMs through Strategy Refinement and Reflection](https://mlsanity.com/paper/2508.13876/) (2025)
- [Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentation](https://mlsanity.com/paper/2506.19998/) (2025)
- [What's the Plan? Evaluating and Developing Planning-Aware Techniques for Language Models](https://mlsanity.com/paper/2402.11489/) (2024)
- [PDDLFuse: A Tool for Generating Diverse Planning Domains](https://mlsanity.com/paper/2411.19886/) (2024)
- [Unifying Inference-Time Planning Language Generation](https://mlsanity.com/paper/2505.14763/) (2025)
- [Language Model as Planner and Formalizer under Constraints](https://mlsanity.com/paper/2510.05486/) (2025)
- [A Survey on Complex Tasks for Goal-Directed Interactive Agents](https://mlsanity.com/paper/2409.18538/) (2024)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [On the Ability of Transformers to Verify Plans](https://mlsanity.com/paper/2603.19954/) (2026)
- [Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results](https://mlsanity.com/paper/2503.13520/) (2025)
- [Iterative Formalization and Planning in Partially Observable Environments](https://mlsanity.com/paper/2505.13126/) (2025)
- [Towards Reliable and Robust LLM Planning: Symbolic Feedback-Driven Iterative Self-Refinement Framework](https://mlsanity.com/paper/2606.27757/) (2026)
- [What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities](https://mlsanity.com/paper/2607.11197/) (2026)
- [Make Planning Research Rigorous Again!](https://mlsanity.com/paper/2505.21674/) (2025)
- [Grounded Evaluation and Repair for NL-to-PDDL Problem Generation](https://mlsanity.com/paper/2609.09898/) (2026)
- [Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms](https://mlsanity.com/paper/2609.27321/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have revolutionized a large variety of NLP tasks. An active debate is to what extent they can do reasoning and planning. Prior work has assessed the latter in the specific context of PDDL planning, based on manually converting three PDDL domains into natural language (NL) prompts. Here we automate this conversion step, showing how to leverage an LLM to automatically generate NL prompts from PDDL input. Our automatically generated NL prompts result in similar LLM-planning performance as the previous manually generated ones. Beyond this, the automation enables us to run much larger experiments, providing for the first time a broad evaluation of LLM planning performance in PDDL. Our NL prompts yield better performance than PDDL prompts and simple template-based NL prompts. Compared to symbolic planners, LLM planning lags far behind; but in some domains, our best LLM configuration scales up further than A^star using LM-cut.