# OLAPH: Improving Factuality in Biomedical Long-form Question Answering

arXiv: https://arxiv.org/abs/2405.12701 · HTML: https://mlsanity.com/paper/2405.12701/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 75 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 30 of 63)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)

## Built on by (most cited first, 20 of 20)

- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs](https://mlsanity.com/paper/2503.02846/) (2025)
- [Automatic Evaluation of Healthcare LLMs Beyond Question-Answering](https://mlsanity.com/paper/2502.06666/) (2025)
- [LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction](https://mlsanity.com/paper/2408.12249/) (2024)
- [Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting](https://mlsanity.com/paper/2503.04013/) (2025)
- [Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality](https://mlsanity.com/paper/2506.20978/) (2025)
- [Structured Outputs Enable General-Purpose LLMs to be Medical Experts](https://mlsanity.com/paper/2503.03194/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.03004/) (2025)
- [Understanding the Effects of Domain Finetuning on LLMs](https://mlsanity.com/paper/2510.09359/) (2025)
- [HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Making](https://mlsanity.com/paper/2409.10011/) (2024)
- [Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs](https://mlsanity.com/paper/2507.16951/) (2025)
- [Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning](https://mlsanity.com/paper/2606.31825/) (2026)
- [Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness](https://mlsanity.com/paper/2607.19322/) (2026)
- [EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering](https://mlsanity.com/paper/2501.12746/) (2025)
- [Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI](https://mlsanity.com/paper/2505.10472/) (2025)
- [MedBioRAG: Semantic Search and Retrieval-Augmented Generation with Large Language Models for Medical and Biological QA](https://mlsanity.com/paper/2512.10996/) (2025)
- [Multi-LLM Adaptive Conformal Inference for Reliable LLM Responses](https://mlsanity.com/paper/2602.01285/) (2026)
- [Culinary Class Wars: Evaluating LLMs using ASH in Cuisine Transfer Task](https://mlsanity.com/paper/2411.01996/) (2024)
- [OpenBioRQ: Unsolved Biomedical Research Questions for Agents](https://mlsanity.com/paper/2606.21959/) (2026)

## Abstract (arXiv)

In the medical domain, numerous scenarios necessitate the long-form generation ability of large language models (LLMs). Specifically, when addressing patients' questions, it is essential that the model's response conveys factual claims, highlighting the need for an automated method to evaluate those claims. Thus, we introduce MedLFQA, a benchmark dataset reconstructed using long-form question-answering datasets related to the biomedical domain. We use MedLFQA to facilitate a cost-effective automatic evaluations of factuality. We also propose OLAPH, a simple and novel framework that utilizes cost-effective and multifaceted automatic evaluation to construct a synthetic preference set and answers questions in our preferred manner. Our framework leads us to train LLMs step-by-step to reduce hallucinations and include crucial medical claims. We highlight that, even on evaluation metrics not used during training, LLMs trained with our OLAPH framework demonstrate significant performance improvement in factuality. Our findings reveal that a 7B LLM trained with our OLAPH framework can provide long answers comparable to the medical experts' answers in terms of factuality. We believe that our work could shed light on gauging the long-text generation ability of LLMs in the medical domain. Our code and datasets are available.