# Hybrid Decision Making via Conformal VLM-generated Guidance

arXiv: https://arxiv.org/abs/2604.14980 · HTML: https://mlsanity.com/paper/2604.14980/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Regression Under Human Assistance](https://mlsanity.com/paper/1909.02963/) (2019)
- [Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosis](https://mlsanity.com/paper/2108.07392/) (2021)
- [Learning To Guide Human Decision Makers With Vision-Language Models](https://mlsanity.com/paper/2403.16501/) (2024)
- [MedGellan: LLM-Generated Medical Guidance to Support Physicians](https://mlsanity.com/paper/2507.04431/) (2025)

## Abstract (arXiv)

Building on recent advances in AI, hybrid decision making (HDM) holds the promise of improving human decision quality and reducing cognitive load. We work in the context of learning to guide (LtG), a recently proposed HDM framework in which the human is always responsible for the final decision: rather than suggesting decisions, in LtG the AI supplies (textual) guidance useful for facilitating decision making. One limiting factor of existing approaches is that their guidance compounds information about all possible outcomes, and as a result it can be difficult to digest. We address this issue by introducing ConfGuide, a novel LtG approach that generates more succinct and targeted guidance. To this end, it employs conformal risk control to select a set of outcomes, ensuring a cap on the false negative rate. We demonstrate our approach on a real-world multi-label medical diagnosis task. Our empirical evaluation highlights the promise of ConfGuide.