mlsanityData to 7 Oct 2026

LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks

arXiv 2603.22744 ↗First posted 24 Mar 2026cs.AIcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models excel on objectively verifiable tasks such as math and programming, where evaluation reduces to unit tests or a single correct answer. In contrast, real-world enterprise work is often subjective and context-dependent: success hinges on organizational goals, user intent, and the quality of intermediate artifacts produced across long, multi-tool workflows. We introduce LH-Bench, a three-pillar evaluation design that moves beyond binary correctness to score autonomous, long-horizon execution on subjective enterprise tasks. The pillars are: (i) expert-grounded rubrics that give LLM judges the domain context needed to score subjective work, (ii) curated ground-truth artifacts that enable stepwise reward signals (e.g., chapter-level annotation for content tasks), and (iii) pairwise human preference evaluation for convergent validation. We show that domain-authored rubrics provide substantially more reliable evaluation signals than LLM-authored rubrics (kappa = 0.60 vs. 0.46), and that human preference judgments confirm the same top-tier separation (p < 0.05), evidence that expert-grounded evaluation can scale without sacrificing reliability. We release public datasets and report results on two environments: Figma-to-code (33 real .fig tasks against the Figma API via MCP) and Programmatic content (41 courses comprising 183 individually-evaluated chapters on a course platform serving 30+ daily users).

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 31 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

19 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios26 Sep
  2. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation28 Jul
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 16

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  8. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  10. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  11. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  12. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
4 more
  1. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  2. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  3. 2025FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluationcited by 17
  4. 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedbackcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals
  2. 2026Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
  3. 2026EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
  4. 2026Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills
  5. 2026A Framework for Evaluating Agentic Skills at Scale
  6. 2026BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
  7. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
  8. 2026The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI