# Beyond the Rubric: Cultural Misalignment in LLM Benchmarks for Sexual and Reproductive Health

arXiv: https://arxiv.org/abs/2511.17554 · HTML: https://mlsanity.com/paper/2511.17554/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 7 of 7)

- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [Automatic Evaluation of Healthcare LLMs Beyond Question-Answering](https://mlsanity.com/paper/2502.06666/) (2025)
- ["Kya family planning after marriage hoti hai?": Integrating Cultural Sensitivity in an LLM Chatbot for Reproductive Health](https://mlsanity.com/paper/2502.15939/) (2025)
- [Rethinking Evidence Hierarchies in Medical Language Benchmarks: A Critical Evaluation of HealthBench](https://mlsanity.com/paper/2508.00081/) (2025)

## Built on by (most cited first, 2 of 2)

- [GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety](https://mlsanity.com/paper/2605.20203/) (2026)
- [Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions](https://mlsanity.com/paper/2609.07687/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been positioned as having the potential to expand access to health information in the Global South, yet their evaluation remains heavily dependent on benchmarks designed around Western norms. We present insights from a preliminary benchmarking exercise with a chatbot for sexual and reproductive health (SRH) for an underserved community in India. We evaluated using HealthBench, a benchmark for conversational health models by OpenAI. We extracted 637 SRH queries from the dataset and evaluated on the 330 single-turn conversations. Responses were evaluated using HealthBench's rubric-based automated grader, which rated responses consistently low. However, qualitative analysis by trained annotators and public health experts revealed that many responses were actually culturally appropriate and medically accurate. We highlight recurring issues, particularly a Western bias, such as for legal framing and norms (e.g., breastfeeding in public), diet assumptions (e.g., fish safe to eat during pregnancy), and costs (e.g., insurance models). Our findings demonstrate the limitations of current benchmarks in capturing the effectiveness of systems built for different cultural and healthcare contexts. We argue for the development of culturally adaptive evaluation frameworks that meet quality standards while recognizing needs of diverse populations.