# One Analyst Is Not Ground Truth: Grading Agent-Built Financial Models Against Observed Professional Practice

arXiv: https://arxiv.org/abs/2607.24889 · HTML: https://mlsanity.com/paper/2607.24889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 11 of 11)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [Fluid Language Model Benchmarking](https://mlsanity.com/paper/2509.11106/) (2025)
- [From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets](https://mlsanity.com/paper/2605.28359/) (2026)
- [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026)
- [FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks](https://mlsanity.com/paper/2604.05912/) (2026)
- [BlueFin: Benchmarking LLM Agents on Financial Spreadsheets](https://mlsanity.com/paper/2605.30907/) (2026)

## Abstract (arXiv)

Language model agents can now construct complete financial models, but it remains unclear whether they have learned the financial judgment that gives those models meaning. Existing benchmarks often anchor correctness to an expert-authored solution (e.g., numerical targets or detailed rubrics) which introduces a implicit assumption: one expert solution can serve as ground truth. This is appropriate when finance provides a unique answer, but not when judgment is required. Evidence from professional practice challenges this assumption: When financial models built by different analysts for the same company are graded against one another, the median score is only 0.33 under standard tolerances, revealing that single-reference grading confounds professional disagreement with error. We therefore introduce GAUGE, a benchmark that decomposes financial-model evaluation into deterministic checks, rubric judgments and numerical rules. Built from 1,001 professional valuation spanning 922 companies and all 25 GICS industry groups, GAUGE checks mechanical properties deterministically and evaluates judgment-bearing quantities against ranges in professional practice. We then validate GAUGE as a measurement instrument by testing expertise ordering, held-out professional values, and robustness to LM judgements. Our findings show that current LM agents are far better at constructing financial models than at deriving the company-specific assumptions that drive valuation, a gap that even persists after fine-tuning.