# FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents arXiv: https://arxiv.org/abs/2608.11683 ยท HTML: https://mlsanity.com/paper/2608.11683/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. - Builds on 32 of its 44 references that are in the map. - First posted on arXiv 12 Aug 2026. ## Placed in - LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/) ## Builds on (most cited first, 30 of 32) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024) - [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025) - [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025) - [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021) - [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023) - [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025) - [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025) - [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022) - [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024) - [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023) - [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024) - [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025) - [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025) - [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024) - [FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering](https://mlsanity.com/paper/2508.14052/) (2025) - [FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging](https://mlsanity.com/paper/2506.05828/) (2025) - [ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists](https://mlsanity.com/paper/2506.01241/) (2025) - [FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents](https://mlsanity.com/paper/2507.16248/) (2025) - [FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain](https://mlsanity.com/paper/2507.17186/) (2025) - [Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings](https://mlsanity.com/paper/2602.07294/) (2026) - [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026) - [FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality](https://mlsanity.com/paper/2607.12252/) (2026) - [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026) - [MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance](https://mlsanity.com/paper/2605.22664/) (2026) - [FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks](https://mlsanity.com/paper/2604.05912/) (2026) - [Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Research](https://mlsanity.com/paper/2604.21006/) (2026) - [FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents](https://mlsanity.com/paper/2607.19238/) (2026) ## Built on by (most cited first, 4 of 4) - [FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability](https://mlsanity.com/paper/2609.25192/) (2026) - [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026) - [Can Language Models Learn to Forecast Stock Prices](https://mlsanity.com/paper/2609.36914/) (2026) - [Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics](https://mlsanity.com/paper/2610.13000/) (2026) ## Abstract (arXiv) AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.