mlsanityData to 14 Oct 2026

FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

arXiv 2609.24002 ↗First posted 21 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specified: Meta Platforms' "operating income" is 46.75B consolidated but 62.87B for the Family of Apps segment, and each reading is exactly verifiable against the filing. A capable agent should recognize the ambiguity and ask, rather than commit to a plausible but unintended reading. Existing financial benchmarks cannot measure this, because one gold answer per question cannot separate agents that resolve the ambiguity from those that guess the common reading, a blind spot we call the single-gold illusion. We release FinInteract, a bilingual (English/Chinese) benchmark of 173 instances that pairs each question with a default and an intended interpretation across a five-category ambiguity taxonomy, and grades whether an agent elicits the right clarification and then integrates it. Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's accuracy by 3.1 times, confirming the illusion. Beyond it, we find that models answer above 90% once the interpretation is supplied but at most 28.9% when they must elicit it themselves, that targeting is uneven across a taxonomy well powered for entity scope and metric definition and exploratory elsewhere, and that conditioning on the ambiguity category improves resolution at both inference and training time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 31 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

16 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering7 Aug

Other references · 15

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  3. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  5. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  6. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  7. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  8. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
  9. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 65
  10. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  11. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 60
  12. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
3 more
  1. 2025FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Modelscited by 19
  2. 2025CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answeringcited by 15
  3. 2026Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questionscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents
  2. 2026Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States
  3. 2026FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
  4. 2026FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
  5. 2026MoCA-Agent: A Market-of-Claims Code Agent for Financial and Numerical Reasoning
  6. 2026Agentic Retrieval-Augmented Generation for Financial Document Question Answering
  7. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Finance
  8. 2026CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answering