# OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

arXiv: https://arxiv.org/abs/2412.13018 · HTML: https://mlsanity.com/paper/2412.13018/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 19 of 19)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [FinTextQA: A Dataset for Long-form Financial Question Answering](https://mlsanity.com/paper/2405.09980/) (2024)
- [RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge](https://mlsanity.com/paper/2311.08147/) (2023)
- [AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework](https://mlsanity.com/paper/2403.12582/) (2024)
- [ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks](https://mlsanity.com/paper/2310.12516/) (2023)
- [Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings](https://mlsanity.com/paper/2402.17016/) (2024)

## Built on by (most cited first, 29 of 29)

- [R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.17005/) (2025)
- [Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward](https://mlsanity.com/paper/2508.12800/) (2025)
- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain](https://mlsanity.com/paper/2505.17471/) (2025)
- [TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning](https://mlsanity.com/paper/2504.08694/) (2025)
- [SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment](https://mlsanity.com/paper/2509.03934/) (2025)
- [Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Finance](https://mlsanity.com/paper/2503.05185/) (2025)
- [Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning](https://mlsanity.com/paper/2510.26205/) (2025)
- [HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches](https://mlsanity.com/paper/2508.08088/) (2025)
- [SustainableQA: A Comprehensive Question Answering Dataset for Corporate Sustainability and EU Taxonomy Reporting](https://mlsanity.com/paper/2508.03000/) (2025)
- [OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews](https://mlsanity.com/paper/2509.00285/) (2025)
- [Respond Beyond Language: A Benchmark for Video Generation in Response to Realistic User Intents](https://mlsanity.com/paper/2506.01689/) (2025)
- [Revisiting RAG Retrievers: An Information Theoretic Benchmark](https://mlsanity.com/paper/2602.21553/) (2026)
- [Retrieving Minimal and Sufficient Reasoning Subgraphs with Graph Foundation Models for Path-aware GraphRAG](https://mlsanity.com/paper/2603.07179/) (2026)
- [KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance](https://mlsanity.com/paper/2510.05524/) (2025)
- [MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.14252/) (2025)
- [OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments](https://mlsanity.com/paper/2605.18758/) (2026)
- [K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance](https://mlsanity.com/paper/2605.29523/) (2026)
- [ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks](https://mlsanity.com/paper/2510.18455/) (2025)
- [Adaptation of Embedding Models to Financial Filings via LLM Distillation](https://mlsanity.com/paper/2512.08088/) (2025)
- [PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents](https://mlsanity.com/paper/2512.14735/) (2025)
- [AIANO: Enhancing Information Retrieval with AI-Augmented Annotation](https://mlsanity.com/paper/2602.04579/) (2026)
- [QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate](https://mlsanity.com/paper/2603.11650/) (2026)
- [STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator](https://mlsanity.com/paper/2604.24544/) (2026)
- [Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2605.14488/) (2026)
- [SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models](https://mlsanity.com/paper/2605.19357/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis](https://mlsanity.com/paper/2608.23758/) (2026)
- [Enhanced Trading Systems with Intelligent Market Analysis](https://mlsanity.com/paper/2610.13998/) (2026)

## Abstract (arXiv)

As a typical and practical application of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) techniques have gained extensive attention, particularly in vertical domains where LLMs may lack domain-specific knowledge. In this paper, we introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework, including (1) a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios; (2) a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47% acceptance ratio in human evaluations on generated instances; (3) a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline; and (4) robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator. Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets and highlights the performance variations of RAG systems across diverse topics and tasks, revealing significant opportunities for RAG models to improve their capabilities in vertical domains. We open source the code of our benchmark in hrefhttps://github.com/RUC-NLPIR/OmniEvalhttps://github.com/RUC-NLPIR/OmniEval.