mlsanityData to 14 Oct 2026

THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

arXiv 2505.11626 ↗First posted 16 May 2025cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 9 of its 23 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

9 of 23 references.

References · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  7. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  8. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  9. 2024On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluationcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domain
  2. 2026Augmenting Question Answering with A Hybrid RAG Approach
  3. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
  4. 2024Toward Optimal Search and Retrieval for RAG
  5. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
  6. 2025Evaluation of retrieval-based QA on QUEST-LOFT
  7. 2024Semantic Tokens in Retrieval Augmented Generation
  8. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems