# TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

arXiv: https://arxiv.org/abs/2606.01046 · HTML: https://mlsanity.com/paper/2606.01046/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 31 May 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 17 of 17)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planning](https://mlsanity.com/paper/2402.07204/) (2024)
- [TripTailor: A Real-World Benchmark for Personalized Travel Planning](https://mlsanity.com/paper/2508.01432/) (2025)
- [TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning](https://mlsanity.com/paper/2502.20508/) (2025)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning](https://mlsanity.com/paper/2504.08694/) (2025)
- [QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search](https://mlsanity.com/paper/2306.06707/) (2023)
- [Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents](https://mlsanity.com/paper/2506.04649/) (2025)
- [Code2MCP: Transforming Code Repositories into MCP Services](https://mlsanity.com/paper/2509.05941/) (2025)
- [PeroMAS: A Multi-agent System of Perovskite Material Discovery](https://mlsanity.com/paper/2602.13312/) (2026)
- [+Tour: Recommending personalized itineraries for smart tourism](https://mlsanity.com/paper/2502.17345/) (2025)

## Built on by (most cited first, 2 of 2)

- [TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning](https://mlsanity.com/paper/2607.26977/) (2026)
- [Replan, Repair, or Edit? A Unified Empirical Evaluation of Travel Agents for Itinerary Revision under Resource Disruptions](https://mlsanity.com/paper/2609.19654/) (2026)

## Abstract (arXiv)

The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existing benchmarks' limitations: 1) overemphasis on constraint compliance, neglecting multi-dimensional qualities like spatio-temporal cost; 2) datasets lacking real-world authenticity and coverage in key areas (e.g., lodging, transport); and 3) isolated daily plan assessments that miss critical details (e.g., the impact of daily accommodation and visit pacing) needed for entire plan's evaluation. To address this gap, we introduce TravelEval, a realistic and comprehensive benchmark. TravelEval features 1) a novel six-dimensional evaluation framework to holistically assess plans across accuracy, compliance, temporality, spatiality, economy, and utility dimensions; 2) a highly realistic data sandbox with precise accommodation pricing and authentic intercity transportation data; and 3) a simulation-based global evaluation method that emulates complete travel plans with API-integrated geographic information and fine-grained queuing time. Evaluating 12 mainstream approaches with TravelEval reveals several valuable insights, such that LLMs struggle with globally-optimized multi-dimensional planning (especially in spatio-temporal reasoning and budget compliance), and agentic reasoning strategies offer no consistent improvement. Concisely, TravelEval facilitates travel plan evaluation via grounded spatio-temporal emulation and comprehensive metrics, providing a robust foundation for advancing LLM-powered travel planning research and applications.