# Label Critic: Design Data Before Models

arXiv: https://arxiv.org/abs/2411.02753 · HTML: https://mlsanity.com/paper/2411.02753/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Network](https://mlsanity.com/paper/1806.05473/) (2018)
- [AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking](https://mlsanity.com/paper/2407.16697/) (2024)
- [Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation](https://mlsanity.com/paper/2101.02323/) (2021)
- [MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision](https://mlsanity.com/paper/2308.16139/) (2023)
- [Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection](https://mlsanity.com/paper/2305.13282/) (2023)
- [Embracing Massive Medical Data](https://mlsanity.com/paper/2407.04687/) (2024)
- [Learning-Based Quality Control for Cardiac MR Images](https://mlsanity.com/paper/1803.09354/) (2018)

## Built on by (most cited first, 4 of 4)

- [PanTS: The Pancreatic Tumor Segmentation Dataset](https://mlsanity.com/paper/2507.01291/) (2025)
- [RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology](https://mlsanity.com/paper/2605.10761/) (2026)
- [ShapeKit](https://mlsanity.com/paper/2506.24003/) (2025)
- [Beyond Masks: The Case for Medical Image Parsing](https://mlsanity.com/paper/2605.11438/) (2026)

## Abstract (arXiv)

As medical datasets rapidly expand, creating detailed annotations of different body structures becomes increasingly expensive and time-consuming. We consider that requesting radiologists to create detailed annotations is unnecessarily burdensome and that pre-existing AI models can largely automate this process. Following the spirit don't use a sledgehammer on a nut, we find that, rather than creating annotations from scratch, radiologists only have to review and edit errors if the Best-AI Labels have mistakes. To obtain the Best-AI Labels among multiple AI Labels, we developed an automatic tool, called Label Critic, that can assess label quality through tireless pairwise comparisons. Extensive experiments demonstrate that, when incorporated with our developed Image-Prompt pairs, pre-existing Large Vision-Language Models (LVLM), trained on natural images and texts, achieve 96.5% accuracy when choosing the best label in a pair-wise comparison, without extra fine-tuning. By transforming the manual annotation task (30-60 min/scan) into an automatic comparison task (15 sec/scan), we effectively reduce the manual efforts required from radiologists by an order of magnitude. When the Best-AI Labels are sufficiently accurate (81% depending on body structures), they will be directly adopted as the gold-standard annotations for the dataset, with lower-quality AI Labels automatically discarded. Label Critic can also check the label quality of a single AI Label with 71.8% accuracy when no alternatives are available for comparison, prompting radiologists to review and edit if the estimated quality is low (19% depending on body structures).