mlsanityData to 14 Oct 2026

MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors

arXiv 2606.17453 ↗First posted 16 Jun 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model agents are increasingly integrated into map services. Since map services are embedded in everyday-life scenarios rather than professional task settings, users often express their needs informally, resulting in underspecified queries with many unspoken needs, namely, implicit decision factors that are critical for user satisfaction. Although clarification is an effective way to mitigate this issue, it increases user burden in daily interaction, and a capable agent should first proactively recover such factors from available information sources. However, evaluating this ability is challenging. The first challenge is to determine which implicit decision factors are suitable for evaluation. A factor is evaluable only if it affects user acceptance and can be recovered from information available to the agent before it responds. Second, user satisfaction cannot be reliably represented by a single reference answer, requiring a benchmark that converts satisfaction-relevant factors into objective and quantifiable evaluation targets. To address these challenges, we propose a restore-identify-filter framework that reconstructs complete user needs from behavior-chain evidence, identifies implicit decision factors, and retains only those supported by pre-query evidence. Building on this methodology, we construct MapSatisfyBench from large-scale, real-world anonymized user data and annotate ground truth from five dimensions and enables full-chain evaluation of satisfaction-aware map agents. Experiments show that current agents generally perform well on explicit task completion, but remain limited in satisfying implicit decision factors and proactively acquiring the evidence needed for satisfaction-aware decisions. These findings establish MapSatisfyBench as a benchmark for shifting map-agent evaluation from task completion toward satisfaction-aware spatial decision making.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 14 of its 90 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

14 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents2 May
  2. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb

Other references · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  5. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  6. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  7. 2021Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systemscited by 47
  8. 2020Analyzing and Learning from User Interactions for Search Clarificationcited by 23
  9. 2025TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planningcited by 21
  10. 2025Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Taskscited by 10
  11. 2023Schema-Guided User Satisfaction Modeling for Task-Oriented Dialoguescited by 8
  12. 2026MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarioscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild
  2. 2026AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
  3. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
  4. 2025LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue
  5. 2026AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling Agents
  6. 2026GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning
  7. 2026Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
  8. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents