# Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

arXiv: https://arxiv.org/abs/2606.15419 · HTML: https://mlsanity.com/paper/2606.15419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 50 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 11 of 11)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports](https://mlsanity.com/paper/2505.11733/) (2025)
- [MDTeamGPT: A Self-Evolving LLM-based Multi-Agent Framework for Multi-Disciplinary Team Medical Consultation](https://mlsanity.com/paper/2503.13856/) (2025)
- [Instruction Diversity Drives Generalization To Unseen Tasks](https://mlsanity.com/paper/2402.10891/) (2024)
- [Why Chain of Thought Fails in Clinical Text Understanding](https://mlsanity.com/paper/2509.21933/) (2025)
- [Structured Outputs Enable General-Purpose LLMs to be Medical Experts](https://mlsanity.com/paper/2503.03194/) (2025)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs](https://mlsanity.com/paper/2506.12182/) (2025)

## Abstract (arXiv)

Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each other's reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain-of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems.