# MIRA: A Bilingual Benchmark for Medical Information Response Audit

arXiv: https://arxiv.org/abs/2605.28025 · HTML: https://mlsanity.com/paper/2605.28025/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 48 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 11 of 11)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Large language models provide unsafe answers to patient-posed medical questions](https://mlsanity.com/paper/2507.18905/) (2025)
- [Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias](https://mlsanity.com/paper/2405.05506/) (2024)
- [LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users](https://mlsanity.com/paper/2406.17737/) (2024)
- [Do LLMs Provide Consistent Answers to Health-Related Questions across Languages?](https://mlsanity.com/paper/2501.14719/) (2025)
- [Susceptibility of Large Language Models to User-Driven Factors in Medical Queries](https://mlsanity.com/paper/2503.22746/) (2025)
- [MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning](https://mlsanity.com/paper/2510.12818/) (2025)
- [Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency](https://mlsanity.com/paper/2510.18556/) (2025)
- [Classroom Final Exam: An Instructor-Tested Reasoning Benchmark](https://mlsanity.com/paper/2602.19517/) (2026)

## Built on by (most cited first, 2 of 2)

- [Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO](https://mlsanity.com/paper/2608.04339/) (2026)
- [Suppressing Pressure, Amplifying Evidence: Self-Guided Attention Steering to Mitigate Sycophancy and Stubbornness](https://mlsanity.com/paper/2610.04329/) (2026)

## Abstract (arXiv)

Existing safety evaluations for large language models overlook whether responses preserve comparable medical information across different user phrasings of the same question. To address this, we introduce the Medical Information Response Audit (MIRA), a bilingual, controlled benchmark that assesses whether LLMs provide comparable medical information across user-side language, register, and health literacy signals. MIRA contains 4,320 prompts built from 60 medically reviewed, low-risk health questions. Across five mainstream LLMs, models answered all medical questions, but responses to low health-literacy signals consistently omitted more key information, provided fewer concrete next steps, and offered less support for independent judgment. We term this pattern Differential Information Dilution (DID). A comparison with 300 real-world health queries provides preliminary evidence of rank-order validity. A knowledge-guided mitigation prompt reduces information dilution for most models, with the largest reductions in underinformative simplification observed for Claude (~8%) and Qwen (~6%). Code and data are available at https://github.com/Rainxu09/MIRA.