FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain
Not on the map. Why?
Abstract (arXiv)
Large Language Models tend to struggle when dealing with specialized domains. While all aspects of evaluation hold importance, factuality is the most critical one. Similarly, reliable fact-checking tools and data sources are essential for hallucination mitigation. We address these issues by providing a comprehensive Fact-checking Benchmark FActBench covering four generation tasks and six state-of-the-art Large Language Models (LLMs) for the Medical domain. We use two state-of-the-art Fact-checking techniques: Chain-of-Thought (CoT) Prompting and Natural Language Inference (NLI). Our experiments show that the fact-checking scores acquired through the Unanimous Voting of both techniques correlate best with Domain Expert Evaluation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 2 Sep 2025.
Builds on
20 of 29 references.
References · 20
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
- 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
- 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
- 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
- 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
8 more
- 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 162
- 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 55
- 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
- 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
- 2024Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checkingcited by 18
- 2023With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulnesscited by 13
- 2024Exploring Factual Entailment with NLI: A News Media Studycited by 4
- 2024Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Outputcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
- 2025Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models
- 2026Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification
- 2025AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
- 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Data
- 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
- 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
- 2025Iterative Tree Analysis for Medical Critics