# Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

arXiv: https://arxiv.org/abs/2605.18419 · HTML: https://mlsanity.com/paper/2605.18419/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence](https://mlsanity.com/paper/2410.11199/) (2024)
- [In-context learning enables multimodal large language models to classify cancer pathology images](https://mlsanity.com/paper/2403.07407/) (2024)
- [How to Configure Good In-Context Sequence for Visual Question Answering](https://mlsanity.com/paper/2312.01571/) (2023)
- [Taming Diffusion for Dataset Distillation with High Representativeness](https://mlsanity.com/paper/2505.18399/) (2025)
- [Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach](https://mlsanity.com/paper/2502.00577/) (2025)
- [Mimic In-Context Learning for Multimodal Tasks](https://mlsanity.com/paper/2504.08851/) (2025)
- [Realistic Data Enrichment for Robust Image Segmentation in Histopathology](https://mlsanity.com/paper/2304.09534/) (2023)
- [Understanding Silent Failures in Medical Image Classification](https://mlsanity.com/paper/2307.14729/) (2023)
- [URCDM: Ultra-Resolution Image Synthesis in Histopathology](https://mlsanity.com/paper/2407.13277/) (2024)
- [Efficient and Effective In-context Demonstration Selection with Coreset](https://mlsanity.com/paper/2511.08977/) (2025)
- [Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models](https://mlsanity.com/paper/2512.07564/) (2025)

## Built on by (most cited first, 1 of 1)

- [TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning](https://mlsanity.com/paper/2610.12516/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) can couple visual perception with open-ended clinical reasoning, making them attractive for computational histopathology. However, fine-tuning billions of parameters on scarce, expert-annotated pathology data is prohibitive, while in-context learning (ICL), which conditions the VLM on demonstrative image-text pairs without parameter updates, suffers from high sensitivity to which examples are selected and how the query is phrased, producing unreliable diagnostics. Existing selection strategies rely on query-dependent nearest-neighbour retrieval that ignores global data structure, require costly parameter updates, or disregard the joint vision-text embedding geometry of VLMs. We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space. GAUC jointly optimises three objectives: (1) a Maximum Mean Discrepancy term enforcing distributional fidelity between coreset and full dataset, (2) an Effective Mutual Information Difference regulariser bounding performance degradation under prompt paraphrases by exploiting the VLM's joint vision-text alignment, and (3) a predictive-uncertainty (entropy) penalty suppressing ambivalent, hallucination-prone outputs. On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC matches the accuracy of the strongest ICL selection and dataset-distillation baselines while substantially improving calibration, prompt robustness, and hallucination rates, all without a single gradient update.