# Bayesian Deep Learning for Affordance Segmentation in images

arXiv: https://arxiv.org/abs/2303.00871 · HTML: https://mlsanity.com/paper/2303.00871/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Probabilistic Object Detection: Definition and Evaluation](https://mlsanity.com/paper/1811.10800/) (2018)
- [Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection](https://mlsanity.com/paper/1809.06006/) (2018)
- [Uncertainty for Identifying Open-Set Errors in Visual Object Detection](https://mlsanity.com/paper/2104.01328/) (2021)
- [Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2012.12195/) (2020)
- [Deep Affordance-grounded Sensorimotor Object Recognition](https://mlsanity.com/paper/1704.02787/) (2017)
- [Are standard Object Segmentation models sufficient for Learning Affordance Segmentation?](https://mlsanity.com/paper/2107.02095/) (2021)
- [Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth](https://mlsanity.com/paper/2104.14202/) (2021)

## Built on by (most cited first, 13 of 13)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023)
- [Learning Precise Affordances from Egocentric Videos for Robotic Manipulation](https://mlsanity.com/paper/2408.10123/) (2024)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2406.01194/) (2024)
- [Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors](https://mlsanity.com/paper/2505.24103/) (2025)
- [O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Views](https://mlsanity.com/paper/2506.06026/) (2025)
- [Segmenting Object Affordances: Reproducibility and Sensitivity to Scale](https://mlsanity.com/paper/2409.01814/) (2024)
- [ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?](https://mlsanity.com/paper/2412.10050/) (2024)
- [Visual Affordance Prediction: Survey and Reproducibility](https://mlsanity.com/paper/2505.05074/) (2025)
- [Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2602.14837/) (2026)
- [Informative Text-Image Alignment for Visual Affordance Learning with Foundation Models](https://mlsanity.com/paper/2509.17074/) (2025)
- [Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers](https://mlsanity.com/paper/2605.03614/) (2026)

## Abstract (arXiv)

Affordances are a fundamental concept in robotics since they relate available actions for an agent depending on its sensory-motor capabilities and the environment. We present a novel Bayesian deep network to detect affordances in images, at the same time that we quantify the distribution of the aleatoric and epistemic variance at the spatial level. We adapt the Mask-RCNN architecture to learn a probabilistic representation using Monte Carlo dropout. Our results outperform the state-of-the-art of deterministic networks. We attribute this improvement to a better probabilistic feature space representation on the encoder and the Bayesian variability induced at the mask generation, which adapts better to the object contours. We also introduce the new Probability-based Mask Quality measure that reveals the semantic and spatial differences on a probabilistic instance segmentation model. We modify the existing Probabilistic Detection Quality metric by comparing the binary masks rather than the predicted bounding boxes, achieving a finer-grained evaluation of the probabilistic segmentation. We find aleatoric variance in the contours of the objects due to the camera noise, while epistemic variance appears in visual challenging pixels.