# A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)

arXiv: https://arxiv.org/abs/2607.11214 · HTML: https://mlsanity.com/paper/2607.11214/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 13 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Active label cleaning for improved dataset quality under resource constraints](https://mlsanity.com/paper/2109.00574/) (2021)
- [Weighted Intersection over Union (wIoU) for Evaluating Image Segmentation](https://mlsanity.com/paper/2107.09858/) (2021)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice](https://mlsanity.com/paper/2206.06484/) (2022)
- [Noisy Annotations in Semantic Segmentation](https://mlsanity.com/paper/2406.10891/) (2024)
- [USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations](https://mlsanity.com/paper/2209.13008/) (2022)
- [Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall](https://mlsanity.com/paper/2309.13938/) (2023)

## Abstract (arXiv)

Labels are critical for both training and evaluating deep learning segmentation models, but are often inconsistent, noisy, or ambiguous at class boundaries. Many approaches have been developed to support training models on weak labels, but few to none currently exist to facilitate evaluating models on unreliable labels. We therefore introduce a method called "Adaptive Resolution Label Aggregation", or "ARLA", which dynamically adapts the resolution of both the label and the model prediction at inference time before the evaluation metrics are computed. We demonstrate how ARLA can be used to better analyse model behaviour with a practical application to a real flood prediction model, where ARLA was able to overcome issues with inconsistent labelling of forested areas and errors in labels within regions of heavy cloud cover. Our work presents a new approach to evaluating segmentation models, with adjustable parameters to adapt the aggregated resolution to the precision of the label or the level of label noise. Fundamentally, ARLA exploits the information encapsulated by a label but minimises the label error, extracting from the noise a clearer signal of a model's true performance.