# MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs

arXiv: https://arxiv.org/abs/2506.22808 · HTML: https://mlsanity.com/paper/2506.22808/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning](https://mlsanity.com/paper/2408.07981/) (2024)
- [JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability](https://mlsanity.com/paper/2402.17887/) (2024)
- [Aloe: A Family of Fine-tuned Open Healthcare LLMs](https://mlsanity.com/paper/2405.01886/) (2024)
- [Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds](https://mlsanity.com/paper/2403.06609/) (2024)

## Built on by (most cited first, 8 of 8)

- [From RAG to Agentic RAG for Faithful Islamic Question Answering](https://mlsanity.com/paper/2601.07528/) (2026)
- [PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics](https://mlsanity.com/paper/2601.03578/) (2026)
- [Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2602.14564/) (2026)
- [What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models](https://mlsanity.com/paper/2605.18738/) (2026)
- [Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs](https://mlsanity.com/paper/2510.20001/) (2025)
- [PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment](https://mlsanity.com/paper/2508.05132/) (2025)
- [Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution](https://mlsanity.com/paper/2605.27621/) (2026)
- [FOCUS: Decoupling Expert Personas in LLMs to Enhance Domain Expert Capabilities](https://mlsanity.com/paper/2608.05611/) (2026)

## Abstract (arXiv)

While Medical Large Language Models (MedLLMs) have demonstrated remarkable potential in clinical tasks, their ethical safety remains insufficiently explored. This paper introduces MedEthicsQA, a comprehensive benchmark comprising 5,623 multiple-choice questions and 5,351 open-ended questions for evaluation of medical ethics in LLMs. We systematically establish a hierarchical taxonomy integrating global medical ethical standards. The benchmark encompasses widely used medical datasets, authoritative question banks, and scenarios derived from PubMed literature. Rigorous quality control involving multi-stage filtering and multi-faceted expert validation ensures the reliability of the dataset with a low error rate (2.72%). Evaluation of state-of-the-art MedLLMs exhibit declined performance in answering medical ethics questions compared to their foundation counterparts, elucidating the deficiencies of medical ethics alignment. The dataset, registered under CC BY-NC 4.0 license, is available at https://github.com/JianhuiWei7/MedEthicsQA.