# ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

arXiv: https://arxiv.org/abs/2510.18941 · HTML: https://mlsanity.com/paper/2510.18941/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 12 of 12)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Search-Time Data Contamination](https://mlsanity.com/paper/2508.13180/) (2025)
- [EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving](https://mlsanity.com/paper/2509.17677/) (2025)

## Built on by (most cited first, 24 of 24)

- [RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation](https://mlsanity.com/paper/2601.08430/) (2026)
- [RubricBench: Aligning Model-Generated Rubrics with Human Standards](https://mlsanity.com/paper/2603.01562/) (2026)
- [Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric](https://mlsanity.com/paper/2602.14069/) (2026)
- [Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction](https://mlsanity.com/paper/2512.14865/) (2025)
- [Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers](https://mlsanity.com/paper/2606.12507/) (2026)
- [Qworld: Question-Specific Evaluation Criteria for LLMs](https://mlsanity.com/paper/2603.23522/) (2026)
- [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026)
- [Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction](https://mlsanity.com/paper/2607.12835/) (2026)
- [Visual Preference Optimization with Rubric Rewards](https://mlsanity.com/paper/2604.13029/) (2026)
- [GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation](https://mlsanity.com/paper/2608.29856/) (2026)
- [Reinforcement Learning with Robust Rubric Rewards](https://mlsanity.com/paper/2605.30244/) (2026)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows](https://mlsanity.com/paper/2604.11304/) (2026)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants](https://mlsanity.com/paper/2606.12608/) (2026)
- [Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation](https://mlsanity.com/paper/2608.24138/) (2026)
- [Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps](https://mlsanity.com/paper/2605.17554/) (2026)
- [RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges](https://mlsanity.com/paper/2608.24231/) (2026)
- [ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation](https://mlsanity.com/paper/2608.22559/) (2026)
- [OSWorld-Pro: Process-based Evaluation for Computer Use Agents](https://mlsanity.com/paper/2609.24890/) (2026)
- [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026)
- [Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement](https://mlsanity.com/paper/2610.02492/) (2026)
- [MA-Bench: Measuring Operational Planning in Language Models](https://mlsanity.com/paper/2610.19138/) (2026)
- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks. Data: https://huggingface.co/datasets/nvidia/ProfBench and Code: https://github.com/NVlabs/ProfBench and Leaderboard: https://huggingface.co/spaces/nvidia/ProfBench