xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 7 Sep 2026.
Builds on
29 of 34 references.
References · 29
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
- 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
- 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
17 more
- 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
- 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
- 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
- 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
- 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
- 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
- 2026Kimi K3: Open Frontier Intelligencecited by 99
- 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 81
- 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
- 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 56
- 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 36
- 2026APEX-Agentscited by 29
- 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contextscited by 26
- 2026Agents' Last Examcited by 25
- 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
- 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarioscited by 8
- 2026SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?cited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents
- 2025ACEBench: Who Wins the Match Point in Tool Usage?
- 2023AgentBench: Evaluating LLMs as Agents
- 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
- 2024Learning to Ask: When LLM Agents Meet Unclear Instruction
- 2026Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
- 2023Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub