Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace. We ask whether such committees can be gamed by shortcuts, cues a benchmark rewards but a clinician would ignore. Across seven cohorts on six public datasets spanning text (MedQA-USMLE, MedMCQA, MIMIC-CXR reports), imaging (NIH ChestX-ray14, MIMIC-CXR-JPG, CheXpert) and tabular ICU records (SUPPORT2), Gemini committees resist these cues in isolation (flip 5-16%), yet a socially plausible shortcut spreads: when two peers assert the same wrong answer, the holdout under test adopts it in 38% of cases, as does a false "pre-screen" system flag, on both capability tiers. Of three oversight agents, a gate cannot separate adoption from honest agreement (false-positive rate 100%); a same-lineage judge reading only the transcript flags adoption on text (precision 100%, recall 93%) but collapses onto the gate in imaging; a referee that privately re-queries the holdout transfers to imaging (77-88% precision, 13-21% false-positive rate). Tripling a cue's visual salience does not move contagion, whereas a second peer voice raises it by half again. Gaming a hidden rubric is near-silent: only 1/10 text and 1/134 imaging drifters name the rubric they moved toward. What games a committee is social plausibility, and only a referee independent of self-report catches it. Code: https://github.com/criticaldata/benchmaxxing
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 24 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
15 of 24 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2016Concrete Problems in AI Safetycited by 1,582
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
- 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
- 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
- 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
- 2025Multi-Agent Risks from Advanced AIcited by 150
- 2024Emergent social conventions and collective bias in LLM populationscited by 115
- 2021Algorithmic Monoculture and Social Welfarecited by 98
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias
- 2026ClinicalAgents: Multi-Agent Orchestration for Clinical Decision Making with Dual-Memory
- 2026From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning
- 2026Faithful Where It Can Be Checked: Auditing a Reflection Agent Against Its System Prompt in a Randomized Trial
- 2026Social Circuits behind Multi-agent Echo Chambers
- 2026One Panel Does Not Fit All: Case-Adaptive Multi-Agent Deliberation for Clinical Prediction
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- 2026Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models