mlsanityData to 14 Oct 2026

Evaluating Biases in Context-Dependent Health Questions

arXiv 2403.04858 ↗First posted 7 Mar 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Chat-based large language models have the opportunity to empower individuals lacking high-quality healthcare access to receive personalized information across a variety of topics. However, users may ask underspecified questions that require additional context for a model to correctly answer. We study how large language model biases are exhibited through these contextual questions in the healthcare domain. To accomplish this, we curate a dataset of sexual and reproductive healthcare questions that are dependent on age, sex, and location attributes. We compare models' outputs with and without demographic context to determine group alignment among our contextual questions. Our experiments reveal biases in each of these attributes, where young adult female users are favored.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 11 of its 22 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

11 of 22 references.

References · 11

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  4. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  5. 2020How Context Affects Language Models' Factual Predictionscited by 181
  6. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  7. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
  8. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  9. 2020Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddingscited by 79
  10. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 57
  11. 2021Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Managementcited by 16

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Reducing Large Language Model Safety Risks in Women's Health using Semantic Entropycited by 8
  2. 2024The Generation Gap: Exploring Age Bias in the Value Systems of Large Language Modelscited by 5
  3. 2025Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AIcited by 1
  4. 2025Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzlescited by 1