# Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

arXiv: https://arxiv.org/abs/2602.07294 · HTML: https://mlsanity.com/paper/2602.07294/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance](https://mlsanity.com/paper/2210.08817/) (2022)
- [Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance](https://mlsanity.com/paper/2502.08127/) (2025)
- [FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging](https://mlsanity.com/paper/2506.05828/) (2025)
- [SEC-QA: A Systematic Evaluation Corpus for Financial QA](https://mlsanity.com/paper/2406.14394/) (2024)
- [FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.15800/) (2025)
- [FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance](https://mlsanity.com/paper/2508.05201/) (2025)
- [SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities](https://mlsanity.com/paper/2504.04596/) (2025)
- [Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models](https://mlsanity.com/paper/2411.06272/) (2024)

## Built on by (most cited first, 8 of 8)

- [FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents](https://mlsanity.com/paper/2608.11683/) (2026)
- [Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines](https://mlsanity.com/paper/2603.08704/) (2026)
- [TRACE: Tourism Recommendation with Accountable Citation Evidence](https://mlsanity.com/paper/2605.07677/) (2026)
- [FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings](https://mlsanity.com/paper/2608.07400/) (2026)
- [FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification](https://mlsanity.com/paper/2605.29586/) (2026)
- [MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios](https://mlsanity.com/paper/2606.24950/) (2026)
- [FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA](https://mlsanity.com/paper/2609.19680/) (2026)
- [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026)

## Abstract (arXiv)

With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is associated with increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.