# Region-Normalized DPO for Medical Image Segmentation under Noisy Judges

arXiv: https://arxiv.org/abs/2601.23222 · HTML: https://mlsanity.com/paper/2601.23222/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Generalized Preference Optimization: A Unified Approach to Offline Alignment](https://mlsanity.com/paper/2402.05749/) (2024)
- [Provably Robust DPO: Aligning Language Models with Noisy Feedback](https://mlsanity.com/paper/2403.00409/) (2024)
- [Token-level Direct Preference Optimization](https://mlsanity.com/paper/2404.11999/) (2024)
- [Direct Preference Optimization with an Offset](https://mlsanity.com/paper/2402.10571/) (2024)
- [β-DPO: Direct Preference Optimization with Dynamic β](https://mlsanity.com/paper/2407.08639/) (2024)
- [Annotation-efficient deep learning for automatic medical image segmentation](https://mlsanity.com/paper/2012.04885/) (2020)
- [V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization](https://mlsanity.com/paper/2411.02712/) (2024)
- [Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization](https://mlsanity.com/paper/2407.07880/) (2024)
- [Curriculum Direct Preference Optimization for Diffusion and Consistency Models](https://mlsanity.com/paper/2405.13637/) (2024)
- [Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth](https://mlsanity.com/paper/1702.03407/) (2017)
- [Robust Preference Optimization via Dynamic Target Margins](https://mlsanity.com/paper/2506.03690/) (2025)
- [Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation](https://mlsanity.com/paper/2406.03323/) (2024)
- [Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.04639/) (2025)
- [Quality Assured: Rethinking Annotation Strategies in Imaging AI](https://mlsanity.com/paper/2407.17596/) (2024)
- [SegQC: a segmentation network-based framework for multi-metric segmentation quality control and segmentation error detection in volumetric medical images](https://mlsanity.com/paper/2411.07601/) (2024)
- [Predictive Accuracy-Based Active Learning for Medical Image Segmentation](https://mlsanity.com/paper/2405.00452/) (2024)
- [Quality-aware semi-supervised learning for CMR segmentation](https://mlsanity.com/paper/2009.00584/) (2020)
- [SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation](https://mlsanity.com/paper/2508.02464/) (2025)
- [Model Agnostic Preference Optimization for Medical Image Segmentation](https://mlsanity.com/paper/2512.15009/) (2025)

## Abstract (arXiv)

While dense pixel-wise annotations remain the gold standard for medical image segmentation, they are costly to obtain and limit scalability. In contrast, many deployed systems already produce inexpensive automatic quality-control (QC) signals like model agreement, uncertainty measures, or learned mask-quality scores which can be used for further model training without additional ground-truth annotation. However, these signals can be noisy and biased, making preference-based fine-tuning susceptible to harmful updates. We study Direct Preference Optimization (DPO) for segmentation from such noisy judges using proposals generated by a supervised base segmenter trained on a small labeled set. We find that outcomes depend strongly on how preference pairs are mined: selecting the judge's top-ranked proposal can improve peak performance when the judge is reliable, but can amplify harmful errors under weaker judges. We propose Region-Normalized DPO (RN-DPO), a segmentation-aware objective which normalizes preference updates by the size of the disagreement region between masks, reducing the leverage of harmful comparisons and improving optimization stability. Across two medical datasets and multiple regimes, RN-DPO improves sustained performance and stabilizes preference-based fine-tuning, outperforming standard DPO and strong baselines without requiring additional pixel annotations.