mlsanityData to 14 Oct 2026

PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

arXiv 2511.11562 ↗First posted 14 Nov 2025cs.CL cs.CYcited by 32

Not on the map. Why?

Abstract (arXiv)

Frontier model progress is often measured by academic benchmarks, which offer a limited view of performance in real-world professional contexts. Existing evaluations often fail to assess open-ended, economically consequential tasks in high-stakes domains like Legal and Finance, where practical returns are paramount. To address this, we introduce Professional Reasoning Bench (PRBench), a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it, to our knowledge, the largest public, rubric-based benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed tasks inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the prompts and analyze performance using human-annotated rubric categories. Our analysis shows that models with similar overall scores can diverge significantly on specific capabilities. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete reasoning, highlighting critical gaps in their reliability for professional adoption.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 14 Nov 2025.

Builds on

20 of 36 references.

References · 20

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  4. 2025Humanity's Last Examcited by 491
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  7. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  8. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  9. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
  10. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  11. 2025Reinforcement Learning with Rubric Anchorscited by 113
  12. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
8 more
  1. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  2. 2024Determinants of LLM-assisted Decision-Makingcited by 56
  3. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  4. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domainscited by 40
  5. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 28
  6. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 20
  7. 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citationscited by 15
  8. 2025The AI Productivity Index (APEX)cited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 31.

32 papers.

Sort:

By area:

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  2. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  3. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
  4. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
  5. 2026M^star: Every Task Deserves Its Own Memory Harnesscited by 11
  6. 2026JobBench: Aligning Agent Work With Human Willcited by 10
  7. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
  8. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 7
  9. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
  10. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  11. 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?cited by 4
  12. 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learningcited by 4
20 more
  1. 2026Reinforcement Learning with Robust Rubric Rewardscited by 4
  2. 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflowscited by 2
  3. 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistantscited by 2
  4. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  5. 2026Thomson: Continual Learning of Frontier Models for SovereignAIcited by 2
  6. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  7. 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncologycited by 1
  8. 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbookscited by 1
  9. 2026On Predicting the Post-training Potential of Pre-trained LLMscited by 1
  10. 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneckcited by 1
  11. 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agentscited by 1
  12. 2026NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Modelscited by 0
  13. 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Datacited by 0
  14. 2026Cross-Domain Hybrid OPD for Generalizable Search Agentscited by 0
  15. 2026ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluationcited by 0
  16. 2026WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluationcited by 0
  17. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  18. 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agentscited by 0
  19. 2026Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluationcited by 0
  20. 2026MA-Bench: Measuring Operational Planning in Language Modelscited by 0