mlsanityData to 7 Oct 2026

FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

arXiv 2510.08945 ↗First posted 10 Oct 2025cs.AIcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, and data spread across these modalities in one or more documents; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier to identify potential pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundation models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics, with wider performance gaps in questions relying on multimodal and cross-document information. Human evaluation of our metrics showed average agreement of 4.62 for correctness and 4.53 for hallucination detection on a 1-5 Likert scale (5 indicating "strongly agree").

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 17 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

9 of 17 references.

References · 9

  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  2. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  3. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  4. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  5. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  6. 2025Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraintscited by 46
  7. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challengescited by 19
  8. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Visioncited by 5
  9. 2024The Path To Autonomous Cyber Defensecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  3. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines
  4. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  5. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  6. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  7. 2024Searching for Best Practices in Retrieval-Augmented Generation
  8. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models