# Affordance detection with Dynamic-Tree Capsule Networks

arXiv: https://arxiv.org/abs/2211.05200 · HTML: https://mlsanity.com/paper/2211.05200/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [3D Point Capsule Networks](https://mlsanity.com/paper/1812.10775/) (2018)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Equivariant Point Network for 3D Point Cloud Analysis](https://mlsanity.com/paper/2103.14147/) (2021)
- [Quaternion Equivariant Capsule Networks for 3D Point Clouds](https://mlsanity.com/paper/1912.12098/) (2019)
- [COVID-CAPS: A Capsule Network-based Framework for Identification of COVID-19 cases from X-ray Images](https://mlsanity.com/paper/2004.02696/) (2020)
- [Geometry Sharing Network for 3D Point Cloud Classification and Segmentation](https://mlsanity.com/paper/1912.10644/) (2019)
- [Capsule Routing via Variational Bayes](https://mlsanity.com/paper/1905.11455/) (2019)
- [VideoCapsuleNet: A Simplified Network for Action Detection](https://mlsanity.com/paper/1805.08162/) (2018)
- [Capsules with Inverted Dot-Product Attention Routing](https://mlsanity.com/paper/2002.04764/) (2020)
- [Onfocus Detection: Identifying Individual-Camera Eye Contact from Unconstrained Images](https://mlsanity.com/paper/2103.15307/) (2021)
- [Increasing the adversarial robustness and explainability of capsule networks with γ-capsules](https://mlsanity.com/paper/1812.09707/) (2018)

## Abstract (arXiv)

Affordance detection from visual input is a fundamental step in autonomous robotic manipulation. Existing solutions to the problem of affordance detection rely on convolutional neural networks. However, these networks do not consider the spatial arrangement of the input data and miss parts-to-whole relationships. Therefore, they fall short when confronted with novel, previously unseen object instances or new viewpoints. One solution to overcome such limitations can be to resort to capsule networks. In this paper, we introduce the first affordance detection network based on dynamic tree-structured capsules for sparse 3D point clouds. We show that our capsule-based network outperforms current state-of-the-art models on viewpoint invariance and parts-segmentation of new object instances through a novel dataset we only used for evaluation and it is publicly available from github.com/gipfelen/DTCG-Net. In the experimental evaluation we will show that our algorithm is superior to current affordance detection methods when faced with grasping previously unseen objects thanks to our Capsule Network enforcing a parts-to-whole representation.