mlsanityData to 14 Oct 2026

xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

arXiv 2506.13651 ↗First posted 16 Jun 2025cs.LGcited by 120

Not on the map. Why?

Cited by 88 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

27 of 42 references.

References · 27

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
15 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  5. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  7. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  8. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  9. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  11. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  12. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  13. 2024ARC Prize 2024: Technical Reportcited by 104
  14. 2025Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performancecited by 14
  15. 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 40; 2026: 80.

120 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025Agentic Reinforced Policy Optimizationcited by 172
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  4. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
  5. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 81
  6. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
  7. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  8. 2025StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?cited by 44
  9. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  10. 2026Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interactioncited by 38
  11. 2026OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesiscited by 37
  12. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 32
38 more
  1. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 26
  2. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  3. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  4. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  5. 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
  6. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  7. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 22
  8. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 22
  9. 2025Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Windowcited by 19
  10. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 18
  11. 2025SCOPE: Prompt Evolution for Enhancing Agent Effectivenesscited by 18
  12. 2025AWorld: Orchestrating the Training Recipe for Agentic AIcited by 17
  13. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  14. 2025BrowseMaster: Towards Scalable Web Browsing via Tool-Augmented Programmatic Agent Paircited by 16
  15. 2026Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Designcited by 16
  16. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  17. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  18. 2026Exploring Reasoning Reward Model for Agentscited by 15
  19. 2026MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Taskscited by 15
  20. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  21. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMscited by 14
  22. 2026Verified Critical Step Optimization for LLM Agentscited by 14
  23. 2026DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivitycited by 14
  24. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 14
  25. 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
  26. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  27. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 12
  28. 2025Pushing Test-Time Scaling Limits of Deep Search with Asymmetric Verificationcited by 12
  29. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  30. 2026OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agentscited by 12
  31. 2026RE-TRAC: REcursive TRAjectory Compression for Deep Search Agentscited by 11
  32. 2025VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seekingcited by 10
  33. 2025WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Modelscited by 10
  34. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 10
  35. 2026Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewardscited by 10
  36. 2026LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agentcited by 10
  37. 2026LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agentscited by 10
  38. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarioscited by 9

50 most cited of 120 All 120 (JSON)