mlsanityData to 14 Oct 2026

Segment Anything with Robust Uncertainty-Accuracy Correlation

arXiv 2605.10603 ↗First posted 11 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite strong zero-shot performance, SAM is unreliable under domain shift due to Mask-level Confidence Confusion (MCC), where a single IoU-based mask score fails to reflect pixel-wise reliability near boundaries. Motivated by the contrast between texture-biased shortcuts in neural networks and shape-centric processing in human vision, we model out-of-domain variation as appearance shifts and non-rigid deformations that jointly stress calibration. We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts. RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry, and applies Uncertainty-Accuracy Alignment to ensure uncertainty consistently highlights erroneous pixels even under adversarial perturbations. Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation. Project page: https://hongyouzhou.github.io/ruac/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 64 of its 81 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

64 of 81 references.

References · 64

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023Segment Anythingcited by 8,234
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
38 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  8. 2015Spatial Transformer Networkscited by 1,429
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2025SAM 3: Segment Anything with Conceptscited by 906
  12. 2021Domain Generalization: A Surveycited by 764
  13. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  14. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  15. 2021Domain Generalization with MixStylecited by 566
  16. 2020Rescaling Egocentric Visioncited by 540
  17. 2023Segment Anything in High Qualitycited by 376
  18. 2023Effective Data Augmentation With Diffusion Modelscited by 283
  19. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 234
  20. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 216
  21. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  22. 2021Reviving Iterative Training with Mask Guidance for Interactive Segmentationcited by 163
  23. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  24. 2024Medical SAM 2: Segment medical images as video via Segment Anything Model 2cited by 154
  25. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 144
  26. 2018Evaluating Bayesian Deep Learning Methods for Semantic Segmentationcited by 143
  27. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  28. 2022Uncertainty Modeling for Out-of-Distribution Generalizationcited by 125
  29. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  30. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 90
  31. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  32. 2021DG-Font: Deformable Generative Networks for Unsupervised Font Generationcited by 70
  33. 2022Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentationcited by 70
  34. 2021Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attackscited by 55
  35. 2024Matching Anything by Segmenting Anythingcited by 50
  36. 2021ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenescited by 47
  37. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  38. 2020TrashCan: A Semantically-Segmented Dataset towards Visual Detection of Marine Debriscited by 45

50 most cited of 64 All 64 (JSON)

Built on by

Not cited yet.