mlsanityData to 14 Oct 2026

FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents

arXiv 2507.16248 ↗First posted 22 Jul 2025cs.CLcited by 12

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Recently, AI agents are rapidly evolving in intelligence and widely used in professional research applications, such as STEM, software development, and finance. Among these AI agents, deep research agent is a key category as it can perform long-horizon tasks and solve problems of greater complexity. However, there are few evaluation frameworks and benchmarks that systematically and automatically investigate the capabilities of these research agents. In addition, financial research problems have distinct complexity and subtlety. To fill in the gap, we propose FinResearchBench, which is a logic tree-based Agent-as-a-Judge and targets specifically for the financial research agents. It provides a comprehensive and automatic assessment of the research agents across 7 key types of tasks in the financial research domain. The contributions of this work are two-folded: (1) the first and innovative Agent-as-a-Judge system that extracts the logic tree of the research outcome and uses it as the intermediate information to present a comprehensive, reliable, and robust evaluation; (2) finance-oriented that it covers 70 typical financial research questions, spreading across 7 frequently encountered types of task in the domain.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 48 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

17 of 48 references.

References · 17

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  8. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  10. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  11. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  12. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
5 more
  1. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  2. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  3. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  4. 2025Deep Research Bench: Evaluating AI Web Research Agentscited by 32
  5. 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2026Agent-as-a-Judgecited by 17
  2. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysiscited by 14
  3. 2026FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Qualitycited by 5
  4. 2026FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agentscited by 4
  5. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  6. 2026Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agentcited by 2
  7. 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Taskscited by 2
  8. 2026Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Researchcited by 2
  9. 2026DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Makingcited by 1
  10. 2026Frontier Financial Judgement: Can agents tell what might move a stock?cited by 0
  11. 2026From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolutioncited by 0
  12. 2026Agent Policy-Value Audit: Separating Transition Composition from Event Selection in Financial LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FinDeepForecast: A Live Multi-Agent System for Benchmarking Deep Research Agents in Financial Forecasting
  2. 2026FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction
  3. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
  4. 2026FinanceHarness: Autonomous Financial Deep Research Framework
  5. 2024FinRobot: AI Agent for Equity Research and Valuation with Large Language Models
  6. 2026DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis
  7. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents
  8. 2024FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models