mlsanityData to 14 Oct 2026

MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation

arXiv 2607.14385 ↗First posted 15 Jul 2026cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation, providing no measure of whether a system can distinguish clinically similar presentations requiring different interventions. We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI: 434 expert-authored clinical narratives in 217 pairs across 371 pathologies, evaluated via the Bias Trap Rate (BTR), the conditional probability that a model fails the counterfactual given success on the base case. We propose Evidence-Anchored RAG (EA-RAG), a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective through clinical parameter extraction, coverage auditing, and contrastive sub-queries. Across eight configurations of four frontier LLMs, base accuracy overstates robust accuracy by 16-28 percentage points in every model. EA-RAG achieves 20.3% BTR and 65.0% robust accuracy on Claude Sonnet 4.6, a 5.5 percentage point BTR reduction without degrading base accuracy. The residual 20% BTR confirms that counterfactual robustness in clinical AI remains an open challenge. Keywords: counterfactual evaluation, clinical AI, maternal healthcare, retrieval-augmented generation, diagnostic robustness

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

21 of 25 references.

References · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
9 more
  1. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  2. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  3. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  4. 2023Query2doc: Query Expansion with Large Language Modelscited by 149
  5. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 88
  6. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  7. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  8. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awarenesscited by 11
  9. 2026MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responsescited by 0
  2. 2026A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combinedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning
  2. 2026Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation
  3. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  4. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
  5. 2026mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
  6. 2026MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
  7. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  8. 2026CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation