# Evaluating Large Language Models for Health-related Queries with Presuppositions

arXiv: https://arxiv.org/abs/2312.08800 · HTML: https://mlsanity.com/paper/2312.08800/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 73 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)

## Built on by (most cited first, 12 of 12)

- [MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models](https://mlsanity.com/paper/2409.19492/) (2024)
- [BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities](https://mlsanity.com/paper/2412.07769/) (2024)
- [Enhancing Recommendation Diversity by Re-ranking with Large Language Models](https://mlsanity.com/paper/2401.11506/) (2024)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence](https://mlsanity.com/paper/2509.04499/) (2025)
- [MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platform](https://mlsanity.com/paper/2506.00308/) (2025)
- [Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation](https://mlsanity.com/paper/2510.16829/) (2025)
- [Zoom In Disparities in Healthcare LLM Q&A](https://mlsanity.com/paper/2510.17476/) (2025)
- [XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?](https://mlsanity.com/paper/2609.06842/) (2026)
- [Assessing Web Search Credibility and Response Groundedness in Chat Assistants](https://mlsanity.com/paper/2510.13749/) (2025)
- [MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication](https://mlsanity.com/paper/2601.09853/) (2026)
- [Towards Emotion Consistency Analysis of Large Language Models in Emotional Conversational Contexts](https://mlsanity.com/paper/2605.06476/) (2026)

## Abstract (arXiv)

As corporations rush to integrate large language models (LLMs) to their search offerings, it is critical that they provide factually accurate information that is robust to any presuppositions that a user may express. In this work, we introduce UPHILL, a dataset consisting of health-related queries with varying degrees of presuppositions. Using UPHILL, we evaluate the factual accuracy and consistency of InstructGPT, ChatGPT, and BingChat models. We find that while model responses rarely disagree with true health claims (posed as questions), they often fail to challenge false claims: responses from InstructGPT agree with 32% of the false claims, ChatGPT 26% and BingChat 23%. As we increase the extent of presupposition in input queries, the responses from InstructGPT and ChatGPT agree with the claim considerably more often, regardless of its veracity. Responses from BingChat, which rely on retrieved webpages, are not as susceptible. Given the moderate factual accuracy, and the inability of models to consistently correct false assumptions, our work calls for a careful assessment of current LLMs for use in high-stakes scenarios.