# Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning

arXiv: https://arxiv.org/abs/2405.20625 · HTML: https://mlsanity.com/paper/2405.20625/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks](https://mlsanity.com/paper/2402.08115/) (2024)
- [Chain of Thoughtlessness? An Analysis of CoT in Planning](https://mlsanity.com/paper/2405.04776/) (2024)
- [Can Large Language Models Reason and Plan?](https://mlsanity.com/paper/2403.04121/) (2024)
- [Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models](https://mlsanity.com/paper/2308.00675/) (2023)
- [Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?](https://mlsanity.com/paper/2401.05302/) (2024)

## Built on by (most cited first, 30 of 34)

- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [PaSa: An LLM Agent for Comprehensive Academic Paper Search](https://mlsanity.com/paper/2501.10120/) (2025)
- [Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces](https://mlsanity.com/paper/2410.09918/) (2024)
- [LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models](https://mlsanity.com/paper/2503.18971/) (2025)
- [Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming](https://mlsanity.com/paper/2410.12112/) (2024)
- [TripTailor: A Real-World Benchmark for Personalized Travel Planning](https://mlsanity.com/paper/2508.01432/) (2025)
- [TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning](https://mlsanity.com/paper/2502.20508/) (2025)
- [Revealing the Barriers of Language Agents in Planning](https://mlsanity.com/paper/2410.12409/) (2024)
- [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025)
- [RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use](https://mlsanity.com/paper/2509.06980/) (2025)
- [TRIP-PAL: Travel Planning with Guarantees by Combining Large Language Models and Automated Planners](https://mlsanity.com/paper/2406.10196/) (2024)
- [Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planning](https://mlsanity.com/paper/2505.16557/) (2025)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [TCP: a Benchmark for Temporal Constraint-Based Planning](https://mlsanity.com/paper/2505.19927/) (2025)
- [TripTide: A Benchmark for Adaptive Travel Planning under Disruptions](https://mlsanity.com/paper/2510.21329/) (2025)
- [LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models](https://mlsanity.com/paper/2502.02441/) (2025)
- [Procedural Knowledge Improves Agentic LLM Workflows](https://mlsanity.com/paper/2511.07568/) (2025)
- [Decompose, Plan in Parallel, and Merge: A Novel Paradigm for Large Language Models based Planning with Multiple Constraints](https://mlsanity.com/paper/2506.02683/) (2025)
- [Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions](https://mlsanity.com/paper/2503.02238/) (2025)
- [We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourism](https://mlsanity.com/paper/2509.11118/) (2025)
- [GRS-QA -- Graph Reasoning-Structured Question Answering Dataset](https://mlsanity.com/paper/2411.00369/) (2024)
- [Can LLMs plan paths with extra hints from solvers?](https://mlsanity.com/paper/2410.05045/) (2024)
- [TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning](https://mlsanity.com/paper/2607.26977/) (2026)
- [Narrative-Driven Travel Planning: Geoculturally-Grounded Script Generation with Evolutionary Itinerary Optimization](https://mlsanity.com/paper/2502.14456/) (2025)
- [Collaborative Problem-Solving in an Optimization Game](https://mlsanity.com/paper/2505.15490/) (2025)
- [TriFlow: A Progressive Multi-Agent Framework for Intelligent Trip Planning](https://mlsanity.com/paper/2512.11271/) (2025)
- [Decoupled Travel Planning with Behavior Forest](https://mlsanity.com/paper/2604.21354/) (2026)
- [Towards Large Language Models for Lunar Mission Planning and In Situ Resource Utilization](https://mlsanity.com/paper/2504.20125/) (2025)
- [CompassLLM: A Multi-Agent Approach toward Geo-Spatial Reasoning for Popular Path Query](https://mlsanity.com/paper/2510.07516/) (2025)
- [ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models](https://mlsanity.com/paper/2603.19515/) (2026)

## Abstract (arXiv)

As the applicability of Large Language Models (LLMs) extends beyond traditional text processing tasks, there is a burgeoning interest in their potential to excel in planning and reasoning assignments, realms traditionally reserved for System 2 cognitive competencies. Despite their perceived versatility, the research community is still unraveling effective strategies to harness these models in such complex domains. The recent discourse introduced by the paper on LLM Modulo marks a significant stride, proposing a conceptual framework that enhances the integration of LLMs into diverse planning and reasoning activities. This workshop paper delves into the practical application of this framework within the domain of travel planning, presenting a specific instance of its implementation. We are using the Travel Planning benchmark by the OSU NLP group, a benchmark for evaluating the performance of LLMs in producing valid itineraries based on user queries presented in natural language. While popular methods of enhancing the reasoning abilities of LLMs such as Chain of Thought, ReAct, and Reflexion achieve a meager 0%, 0.6%, and 0% with GPT3.5-Turbo respectively, our operationalization of the LLM-Modulo framework for TravelPlanning domain provides a remarkable improvement, enhancing baseline performances by 4.6x for GPT4-Turbo and even more for older models like GPT3.5-Turbo from 0% to 5%. Furthermore, we highlight the other useful roles of LLMs in the planning pipeline, as suggested in LLM-Modulo, which can be reliably operationalized such as extraction of useful critics and reformulator for critics.