PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
Not on the map. Why?
Abstract (arXiv)
Frontier model progress is often measured by academic benchmarks, which offer a limited view of performance in real-world professional contexts. Existing evaluations often fail to assess open-ended, economically consequential tasks in high-stakes domains like Legal and Finance, where practical returns are paramount. To address this, we introduce Professional Reasoning Bench (PRBench), a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it, to our knowledge, the largest public, rubric-based benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed tasks inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the prompts and analyze performance using human-annotated rubric categories. Our analysis shows that models with similar overall scores can diverge significantly on specific capabilities. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete reasoning, highlighting critical gaps in their reliability for professional adoption.
Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 14 Nov 2025.
Builds on
20 of 36 references.
References · 20
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2025Humanity's Last Examcited by 491
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
- 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
- 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
- 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
- 2025Reinforcement Learning with Rubric Anchorscited by 113
- 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
8 more
- 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
- 2024Determinants of LLM-assisted Decision-Makingcited by 56
- 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
- 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domainscited by 40
- 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 28
- 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 20
- 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citationscited by 15
- 2025The AI Productivity Index (APEX)cited by 10
Built on by
Citing papers per year: 2025: 1; 2026: 31.
32 papers.
Sort:
By area:
- 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
- 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
- 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
- 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
- 2026M^star: Every Task Deserves Its Own Memory Harnesscited by 11
- 2026JobBench: Aligning Agent Work With Human Willcited by 10
- 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
- 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 7
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
- 2026Visual Preference Optimization with Rubric Rewardscited by 5
- 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?cited by 4
- 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learningcited by 4
20 more
- 2026Reinforcement Learning with Robust Rubric Rewardscited by 4
- 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflowscited by 2
- 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistantscited by 2
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
- 2026Thomson: Continual Learning of Frontier Models for SovereignAIcited by 2
- 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
- 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncologycited by 1
- 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbookscited by 1
- 2026On Predicting the Post-training Potential of Pre-trained LLMscited by 1
- 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneckcited by 1
- 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agentscited by 1
- 2026NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Modelscited by 0
- 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Datacited by 0
- 2026Cross-Domain Hybrid OPD for Generalizable Search Agentscited by 0
- 2026ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluationcited by 0
- 2026WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluationcited by 0
- 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
- 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agentscited by 0
- 2026Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluationcited by 0
- 2026MA-Bench: Measuring Operational Planning in Language Modelscited by 0
- 2026MA-Bench: Measuring Operational Planning in Language Models13 Oct 2026
- 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents30 Sep 2026
- 2026Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation29 Sep 2026
- 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents28 Sep 2026
- 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards27 Sep 2026
- 2026WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation10 Sep 2026
- 2026Thomson: Continual Learning of Frontier Models for SovereignAI27 Aug 2026
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation25 Aug 2026
- 2026ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation23 Aug 2026
- 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck19 Aug 2026
- 2026Cross-Domain Hybrid OPD for Generalizable Search Agents3 Aug 2026
- 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data17 Jul 2026
20 more
- 2026NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models25 Jun 2026
- 2026Autodata: An agentic data scientist to create high quality synthetic data24 Jun 2026
- 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants10 Jun 2026
- 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents2 Jun 2026
- 2026Reinforcement Learning with Robust Rubric Rewards28 May 2026
- 2026JobBench: Aligning Agent Work With Human Will25 May 2026
- 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning22 May 2026
- 2026On Predicting the Post-training Potential of Pre-trained LLMs12 May 2026
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution8 May 2026
- 2026Visual Preference Optimization with Rubric Rewards14 Apr 2026
- 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows13 Apr 2026
- 2026M^star: Every Task Deserves Its Own Memory Harness10 Apr 2026
- 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology2 Mar 2026
- 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?27 Feb 2026
- 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks16 Feb 2026
- 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb 2026
- 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice23 Jan 2026
- 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions21 Jan 2026
- 2026Agentic Rubrics as Contextual Verifiers for SWE Agents7 Jan 2026
- 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?28 Nov 2025
- 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
- 2026M^star: Every Task Deserves Its Own Memory Harnesscited by 11
- 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 7
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
- 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?cited by 4
- 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflowscited by 2
- 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agentscited by 1
- 2026Cross-Domain Hybrid OPD for Generalizable Search Agentscited by 0
- 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents30 Sep 2026
- 2026Cross-Domain Hybrid OPD for Generalizable Search Agents3 Aug 2026
- 2026Autodata: An agentic data scientist to create high quality synthetic data24 Jun 2026
- 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents2 Jun 2026
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution8 May 2026
- 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows13 Apr 2026
- 2026M^star: Every Task Deserves Its Own Memory Harness10 Apr 2026
- 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?27 Feb 2026
- 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
- 2026Reinforcement Learning with Robust Rubric Rewardscited by 4
- 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistantscited by 2
- 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneckcited by 1
- 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck19 Aug 2026
- 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants10 Jun 2026
- 2026Reinforcement Learning with Robust Rubric Rewards28 May 2026
- 2026Agentic Rubrics as Contextual Verifiers for SWE Agents7 Jan 2026
- 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
- 2026JobBench: Aligning Agent Work With Human Willcited by 10
- 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
- 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learningcited by 4
- 2026Thomson: Continual Learning of Frontier Models for SovereignAIcited by 2
- 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
- 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncologycited by 1
- 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbookscited by 1
- 2026On Predicting the Post-training Potential of Pre-trained LLMscited by 1
- 2026NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Modelscited by 0
- 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Datacited by 0
- 2026ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluationcited by 0
12 most cited of 16 All 16 (JSON)
- 2026MA-Bench: Measuring Operational Planning in Language Models13 Oct 2026
- 2026Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation29 Sep 2026
- 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents28 Sep 2026
- 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards27 Sep 2026
- 2026Thomson: Continual Learning of Frontier Models for SovereignAI27 Aug 2026
- 2026ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation23 Aug 2026
- 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data17 Jul 2026
- 2026NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models25 Jun 2026
- 2026JobBench: Aligning Agent Work With Human Will25 May 2026
- 2026ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning22 May 2026
- 2026On Predicting the Post-training Potential of Pre-trained LLMs12 May 2026
- 2026PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology2 Mar 2026
12 newest of 16 All 16 (JSON)