# Enhancing Financial Question Answering with a Multi-Agent Reflection Framework

arXiv: https://arxiv.org/abs/2410.21741 · HTML: https://mlsanity.com/paper/2410.21741/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Solving Math Word Problems via Cooperative Reasoning induced Language Models](https://mlsanity.com/paper/2210.16257/) (2022)
- [Designing Heterogeneous LLM Agents for Financial Sentiment Analysis](https://mlsanity.com/paper/2401.05799/) (2024)
- [TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data](https://mlsanity.com/paper/2401.13223/) (2024)
- [Interpreting and Mitigating Hallucination in MLLMs through Multi-agent Debate](https://mlsanity.com/paper/2407.20505/) (2024)
- [UniRPG: Unified Discrete Reasoning over Table and Text as Program Generation](https://mlsanity.com/paper/2210.08249/) (2022)

## Built on by (most cited first, 12 of 12)

- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk](https://mlsanity.com/paper/2502.15865/) (2025)
- [SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types](https://mlsanity.com/paper/2412.11757/) (2024)
- [Improved LLM Agents for Financial Document Question Answering](https://mlsanity.com/paper/2506.08726/) (2025)
- [Emergent Bias and Fairness in Multi-Agent Decision Systems](https://mlsanity.com/paper/2512.16433/) (2025)
- [MULTITAT: Benchmarking Multilingual Table-and-Text Question Answering](https://mlsanity.com/paper/2502.17253/) (2025)
- [AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification](https://mlsanity.com/paper/2606.03031/) (2026)
- [OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent](https://mlsanity.com/paper/2507.02353/) (2025)
- [What Factors Affect LLMs and RLLMs in Financial Question Answering?](https://mlsanity.com/paper/2507.08339/) (2025)
- [Data-Driven Function Calling Improvements in Large Language Model for Online Financial QA](https://mlsanity.com/paper/2604.05387/) (2026)
- [Synthetic Data-Driven Prompt Tuning for Financial QA over Tables and Documents](https://mlsanity.com/paper/2511.06292/) (2025)
- [Fighting Numerical Hallucinations via Data-centric Compilation for Online Financial QA](https://mlsanity.com/paper/2605.31064/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) have shown impressive capabilities in numerous Natural Language Processing (NLP) tasks, they still struggle with financial question answering (QA), particularly when numerical reasoning is required. Recently, LLM-based multi-agent frameworks have demonstrated remarkable effectiveness in multi-step reasoning, which is crucial for financial QA tasks as it involves extracting relevant information from tables and text and then performing numerical reasoning on the extracted data to infer answers. In this study, we propose a multi-agent framework incorporating a critic agent that reflects on the reasoning steps and final answers for each question. Additionally, we enhance our system by adding multiple critic agents, each focusing on a specific aspect of the answer. Our results indicate that this framework significantly improves performance compared to single-agent reasoning, with an average performance increase of 15% for the LLaMA3-8B model and 5% for the LLaMA3-70B model. Furthermore, our framework performs on par with, and in some cases surpasses, larger single-agent LLMs such as LLaMA3.1-405B and GPT-4o-mini, though it falls slightly short compared to Claude-3.5 Sonnet. Overall, our framework presents an effective solution to enhance open-source LLMs for financial QA tasks, offering a cost-effective alternative to larger models like Claude-3.5 Sonnet.