# Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI

arXiv: https://arxiv.org/abs/2603.20595 · HTML: https://mlsanity.com/paper/2603.20595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [Medical Hallucinations in Foundation Models and Their Impact on Healthcare](https://mlsanity.com/paper/2503.05777/) (2025)
- [RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework](https://mlsanity.com/paper/2408.01262/) (2024)
- [Argumentative Large Language Models for Explainable and Contestable Claim Verification](https://mlsanity.com/paper/2405.02079/) (2024)
- [Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.15734/) (2025)
- [Hierarchical Document Refinement for Long-context Retrieval-augmented Generation](https://mlsanity.com/paper/2505.10413/) (2025)
- [ArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemes](https://mlsanity.com/paper/2403.06294/) (2024)
- [Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework](https://mlsanity.com/paper/2406.14783/) (2024)
- [Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare](https://mlsanity.com/paper/2211.16444/) (2022)
- [SAGE: A Framework of Precise Retrieval for RAG](https://mlsanity.com/paper/2503.01713/) (2025)
- [Metadata-based Data Exploration with Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2410.04231/) (2024)
- [Adaptive Collaboration of Arena-Based Argumentative LLMs for Explainable and Contestable Legal Reasoning](https://mlsanity.com/paper/2602.18916/) (2026)

## Abstract (arXiv)

Multi-agent systems (MAS) are increasingly used in healthcare to support complex decision-making through collaboration among specialized agents. Because these systems act as collective decision-makers, they raise challenges for trust, accountability, and human oversight. Existing approaches to trustworthy AI largely rely on explainability, but explainability alone is insufficient in multi-agent settings, as it does not enable care partners to challenge or correct system outputs. To address this limitation, Contestable AI (CAI) characterizes systems that support effective human challenge throughout the decision-making lifecycle by providing transparency, structured opportunities for intervention, and mechanisms for review, correction, or override. This position paper argues that contestability is a necessary design requirement for trustworthy multi-agent algorithmic care systems. We identify key limitations in current MAS and Explainable AI (XAI) research and present a human-in-the-loop framework that integrates structured argumentation and role-based contestation to preserve human agency, clinical responsibility, and trust in high-stakes care contexts.