# DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents

arXiv: https://arxiv.org/abs/2509.21842 · HTML: https://mlsanity.com/paper/2509.21842/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [AgentSims: An Open-Source Sandbox for Large Language Model Evaluation](https://mlsanity.com/paper/2308.04026/) (2023)
- [AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.08755/) (2025)
- [rStar2-Agent: Agentic Reasoning Technical Report](https://mlsanity.com/paper/2508.20722/) (2025)
- [TravelAgent: An AI Assistant for Personalized Travel Planning](https://mlsanity.com/paper/2409.08069/) (2024)
- [RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning](https://mlsanity.com/paper/2507.07451/) (2025)
- [ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planning](https://mlsanity.com/paper/2402.07204/) (2024)
- [TripTailor: A Real-World Benchmark for Personalized Travel Planning](https://mlsanity.com/paper/2508.01432/) (2025)
- [TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning](https://mlsanity.com/paper/2502.20508/) (2025)
- [TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning](https://mlsanity.com/paper/2504.08694/) (2025)
- [RETAIL: Towards Real-world Travel Planning for Large Language Models](https://mlsanity.com/paper/2508.15335/) (2025)
- [DiMA: An LLM-Powered Ride-Hailing Assistant at DiDi](https://mlsanity.com/paper/2503.04768/) (2025)

## Built on by (most cited first, 17 of 17)

- [Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem](https://mlsanity.com/paper/2512.24873/) (2025)
- [ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking](https://mlsanity.com/paper/2601.06487/) (2026)
- [MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios](https://mlsanity.com/paper/2602.22638/) (2026)
- [Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe](https://mlsanity.com/paper/2603.21972/) (2026)
- [Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling](https://mlsanity.com/paper/2604.08178/) (2026)
- [Agent-Omit: Adaptive Context Omission for Efficient LLM Agents](https://mlsanity.com/paper/2602.04284/) (2026)
- [GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning](https://mlsanity.com/paper/2605.25200/) (2026)
- [TourPlanner: A Competitive Consensus Framework with Constraint-Gated Reinforcement Learning for Travel Planning](https://mlsanity.com/paper/2601.04698/) (2026)
- [HiMAP-Travel: Hierarchical Multi-Agent Planning for Long-Horizon Constrained Travel](https://mlsanity.com/paper/2603.04750/) (2026)
- [VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?](https://mlsanity.com/paper/2608.15265/) (2026)
- [TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward](https://mlsanity.com/paper/2510.09011/) (2025)
- [AMAP Agentic Planning Technical Report](https://mlsanity.com/paper/2512.24957/) (2025)
- [PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models](https://mlsanity.com/paper/2605.20873/) (2026)
- [ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL](https://mlsanity.com/paper/2609.21378/) (2026)
- [Revisiting the Travel Planning Capabilities of Large Language Models](https://mlsanity.com/paper/2605.03308/) (2026)
- [From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.23382/) (2026)
- [Embark Now: User Demand Oriented Framework for Multi-day Urban Travel Itinerary Planning](https://mlsanity.com/paper/2607.10651/) (2026)

## Abstract (arXiv)

Travel planning (TP) agent has recently worked as an emerging building block to interact with external tools/resources for travel itinerary generation, ensuring an enjoyable user experience. Despite its benefits, existing studies rely on hand-craft prompt and fixed agent workflow, hindering more flexible and autonomous TP agents. This paper proposes DeepTravel, an end-to-end agentic reinforcement learning framework for building an autonomous travel planning agent, capable of autonomously planning, executing tools, and reflecting on tool responses to explore, verify, and refine intermediate actions in multi-step reasoning. To achieve this, we first construct a robust travel sandbox by caching transportation, accommodation and POI data, facilitating TP agent training without being constrained by real-world APIs limitations (e.g., inconsistent outputs). Moreover, we develop a hierarchical reward modeling system, where a trajectory-level verifier first checks spatiotemporal feasibility and filters unsatisfied travel itinerary, and then the turn-level verifier further validate itinerary's detail consistency with tool responses, enabling efficient and precise reward service. Finally, we propose the reply-augmented reinforcement learning method that enables TP agent to periodically replay from a failure experience buffer, emerging notable agentic capacity. We deploy the trained TP agent in the DiDi Enterprise Solutions application. A three-month online test shows that it achieves 82% accuracy in travel itinerary generation. Comprehensive offline evaluations further demonstrate that DeepTravel enables small-sized LLMs (e.g., Qwen3-32B) to significantly outperform frontier LLMs (e.g., OpenAI o1/o3 and DeepSeek-R1) and existing TP agent frameworks.