# SMARTFinRAG: Interactive Modularized Financial RAG Benchmark

arXiv: https://arxiv.org/abs/2504.18024 · HTML: https://mlsanity.com/paper/2504.18024/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 6 of 6)

- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [FinArena: A Human-Agent Collaboration Framework for Financial Market Analysis and Forecasting](https://mlsanity.com/paper/2503.02692/) (2025)
- [FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline](https://mlsanity.com/paper/2410.13959/) (2024)

## Abstract (arXiv)

Financial sectors are rapidly adopting language model technologies, yet evaluating specialized RAG systems in this domain remains challenging. This paper introduces SMARTFinRAG, addressing three critical gaps in financial RAG assessment: (1) a fully modular architecture where components can be dynamically interchanged during runtime; (2) a document-centric evaluation paradigm generating domain-specific QA pairs from newly ingested financial documents; and (3) an intuitive interface bridging research-implementation divides. Our evaluation quantifies both retrieval efficacy and response quality, revealing significant performance variations across configurations. The platform's open-source architecture supports transparent, reproducible research while addressing practical deployment challenges faced by financial institutions implementing RAG systems.