# Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy

arXiv: https://arxiv.org/abs/2402.08806 · HTML: https://mlsanity.com/paper/2402.08806/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 3 of 3)

- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance](https://mlsanity.com/paper/2206.11922/) (2022)

## Built on by (most cited first, 2 of 2)

- [Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?](https://mlsanity.com/paper/2603.04421/) (2026)
- [Human-AI collectives produce the most accurate differential diagnoses](https://mlsanity.com/paper/2406.14981/) (2024)

## Abstract (arXiv)

Background: Large language models (LLMs) such as OpenAI's GPT-4 or Google's PaLM 2 are proposed as viable diagnostic support tools or even spoken of as replacements for "curbside consults". However, even LLMs specifically trained on medical topics may lack sufficient diagnostic accuracy for real-life applications. Methods: Using collective intelligence methods and a dataset of 200 clinical vignettes of real-life cases, we assessed and compared the accuracy of differential diagnoses obtained by asking individual commercial LLMs (OpenAI GPT-4, Google PaLM 2, Cohere Command, Meta Llama 2) against the accuracy of differential diagnoses synthesized by aggregating responses from combinations of the same LLMs. Results: We find that aggregating responses from multiple, various LLMs leads to more accurate differential diagnoses (average accuracy for 3 LLMs: 75.3%± 1.6pp) compared to the differential diagnoses produced by single LLMs (average accuracy for single LLMs: 59.0%± 6.1pp). Discussion: The use of collective intelligence methods to synthesize differential diagnoses combining the responses of different LLMs achieves two of the necessary steps towards advancing acceptance of LLMs as a diagnostic support tool: (1) demonstrate high diagnostic accuracy and (2) eliminate dependence on a single commercial vendor.