mlsanityData to 7 Oct 2026

FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

arXiv 2608.11683 ↗First posted 12 Aug 2026cs.AI cs.CLcited by 3

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 44 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

32 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality14 Jul
  2. 2026FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance11 Jul
  3. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents2 Jun
  4. 2026MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance21 May
  5. 2026Herculean: An Agentic Benchmark for Financial Intelligence14 May
  6. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering7 Aug
  7. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents22 Jul

Other references · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  6. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  8. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  9. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  10. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  11. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  12. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
13 more
  1. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  2. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  3. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  4. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 63
  5. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59
  6. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 27
  7. 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 22
  8. 2025FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domaincited by 8
  9. 2026Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filingscited by 8
  10. 2026FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Taskscited by 2
  11. 2026Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Researchcited by 2
  12. 2026FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documentscited by 2
  13. 2026Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Frontier Financial Judgement: Can agents tell what might move a stock?
  2. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents
  3. 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
  4. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
  5. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
  6. 2024FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models
  7. 2026OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents
  8. 2026FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction