mlsanityData to 14 Oct 2026

Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages

arXiv 2407.03321 ↗First posted 3 Jul 2024cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent works have explored using language models for planning problems. One approach examines translating natural language descriptions of planning tasks into structured planning languages, such as the planning domain definition language (PDDL). Existing evaluation methods struggle to ensure semantic correctness and rely on simple or unrealistic datasets. To bridge this gap, we introduce textitPlanetarium, a benchmark designed to evaluate language models' ability to generate PDDL code from natural language descriptions of planning tasks. textitPlanetarium features a novel PDDL equivalence algorithm that flexibly evaluates the correctness of generated PDDL, along with a dataset of 145,918 text-to-PDDL pairs across 73 unique state combinations with varying levels of difficulty. Finally, we evaluate several API-access and open-weight language models that reveal this task's complexity. For example, 96.1% of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, 94.4% are solvable, but only 24.8% are semantically correct, highlighting the need for a more rigorous benchmark for this problem.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

24 of 44 references.

References · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2023Mistral 7Bcited by 3,164
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  9. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  12. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
12 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  2. 2020Unsupervised Translation of Programming Languagescited by 263
  3. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  4. 2019SPoC: Search-based Pseudocode to Codecited by 176
  5. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  6. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  7. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  8. 2021Skill Induction and Planning with Latent Languagecited by 99
  9. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  10. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planningcited by 16
  11. 2023ConSOR: A Context-Aware Semantic Object Rearrangement Framework for Partially Arranged Scenescited by 6
  12. 2021D-VAL: An automatic functional equivalence validation tool for planning domain modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 14.

33 papers.

Sort:

By area:

  1. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  2. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 46
  3. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  4. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  5. 2024Benchmarking Agentic Workflow Generationcited by 37
  6. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  7. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  8. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
  9. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Studycited by 8
  10. 2025R-ConstraintBench: Evaluating LLMs on NP-Complete Schedulingcited by 5
  11. 2025End-to-end PDDL Planning with Hardcoded and Dynamic Agentscited by 5
  12. 2026Using large language models for embodied planning introduces systematic safety riskscited by 5
21 more
  1. 2025Unifying Inference-Time Planning Language Generationcited by 3
  2. 2025Vision Language Models Cannot Plan, but Can They Formalize?cited by 3
  3. 2025Constrained Adaptive Rejection Samplingcited by 3
  4. 2025Language Model as Planner and Formalizer under Constraintscited by 3
  5. 2025An Extensive Evaluation of PDDL Capabilities in off-the-shelf LLMscited by 2
  6. 2025Evaluating Human Trust in LLM-Based Planners: A Preliminary Studycited by 1
  7. 2026AI+HW 2035: Shaping the Next Decadecited by 1
  8. 2026Language Models as Higher-Order Planning Formalizerscited by 1
  9. 2026Model Space Reasoning as Search in Feedback Space for Planning Domain Generationcited by 1
  10. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  11. 2025Make Planning Research Rigorous Again!cited by 0
  12. 2026Textual Planning with Explicit Latent Transitionscited by 0
  13. 2026Text2Model: Modeling Copilots for Text-to-Model Translationcited by 0
  14. 2026Robust Asynchronous Planning via Auto-Formalizationcited by 0
  15. 2026A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Plannerscited by 0
  16. 2026Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agentscited by 0
  17. 2026Reliability-Asymmetric Spacecraft Autonomy: Co-Designing a Capable Learned GNC Stack with a Verified, Adaptation-Aware Runtime Shieldcited by 0
  18. 2026Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinccited by 0
  19. 2026PDDLCoder: Agentic PDDL Generation for LLM-Assisted Symbolic Planningcited by 0
  20. 2026From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planningcited by 0
  21. 2026ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Modelscited by 0