mlsanityData to 14 Oct 2026

TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

arXiv 2606.01046 ↗First posted 31 May 2026cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existing benchmarks' limitations: 1) overemphasis on constraint compliance, neglecting multi-dimensional qualities like spatio-temporal cost; 2) datasets lacking real-world authenticity and coverage in key areas (e.g., lodging, transport); and 3) isolated daily plan assessments that miss critical details (e.g., the impact of daily accommodation and visit pacing) needed for entire plan's evaluation. To address this gap, we introduce TravelEval, a realistic and comprehensive benchmark. TravelEval features 1) a novel six-dimensional evaluation framework to holistically assess plans across accuracy, compliance, temporality, spatiality, economy, and utility dimensions; 2) a highly realistic data sandbox with precise accommodation pricing and authentic intercity transportation data; and 3) a simulation-based global evaluation method that emulates complete travel plans with API-integrated geographic information and fine-grained queuing time. Evaluating 12 mainstream approaches with TravelEval reveals several valuable insights, such that LLMs struggle with globally-optimized multi-dimensional planning (especially in spatio-temporal reasoning and budget compliance), and agentic reasoning strategies offer no consistent improvement. Concisely, TravelEval facilitates travel plan evaluation via grounded spatio-temporal emulation and comprehensive metrics, providing a robust foundation for advancing LLM-powered travel planning research and applications.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 31 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

17 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents5 Jun
  2. 2025TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning11 Apr
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  4. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 13

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2024ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planningcited by 33
  7. 2025TripTailor: A Real-World Benchmark for Personalized Travel Planningcited by 22
  8. 2025TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planningcited by 21
  9. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  10. 2023QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Searchcited by 10
  11. 2025Code2MCP: Transforming Code Repositories into MCP Servicescited by 10
  12. 2026PeroMAS: A Multi-agent System of Perovskite Material Discoverycited by 2
1 more
  1. 2025+Tour: Recommending personalized itineraries for smart tourismcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planningcited by 2
  2. 2026Replan, Repair, or Edit? A Unified Empirical Evaluation of Travel Agents for Itinerary Revision under Resource Disruptionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks
  2. 2026Trip+: Benchmarking Agents in Personalized Interactive Travel Planning
  3. 2026Revisiting the Travel Planning Capabilities of Large Language Models
  4. 2024Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning
  5. 2026GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning
  6. 2025ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning
  7. 2026MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
  8. 2025LLM-ABM for Transportation: Assessing the Potential of LLM Agents in System Analysis