# GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency

arXiv: https://arxiv.org/abs/2412.09511 · HTML: https://mlsanity.com/paper/2412.09511/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts](https://mlsanity.com/paper/2211.05272/) (2022)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [Robo3D: Towards Robust and Reliable 3D Perception against Corruptions](https://mlsanity.com/paper/2303.17597/) (2023)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [AKB-48: A Real-World Articulated Object Knowledge Base](https://mlsanity.com/paper/2202.08432/) (2022)
- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [Affordance Transfer Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2104.02867/) (2021)
- [Learning to Act Properly: Predicting and Explaining Affordances from Images](https://mlsanity.com/paper/1712.07576/) (2017)
- [Benchmarking and Analyzing Point Cloud Classification under Corruptions](https://mlsanity.com/paper/2202.03377/) (2022)
- [Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model](https://mlsanity.com/paper/2401.17868/) (2024)
- [LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding](https://mlsanity.com/paper/2303.09665/) (2023)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)

## Built on by (most cited first, 18 of 18)

- [Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement](https://mlsanity.com/paper/2511.11702/) (2025)
- [Affordance Agent Harness: Verification-Gated Skill Orchestration](https://mlsanity.com/paper/2605.00663/) (2026)
- [Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models](https://mlsanity.com/paper/2605.29074/) (2026)
- [Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation](https://mlsanity.com/paper/2503.03556/) (2025)
- [GarmentX: Autoregressive Parametric Representations for High-Fidelity 3D Garment Generation](https://mlsanity.com/paper/2504.20409/) (2025)
- [Zero-Shot 3D Visual Grounding from Vision-Language Models](https://mlsanity.com/paper/2505.22429/) (2025)
- [Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge](https://mlsanity.com/paper/2510.08316/) (2025)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos](https://mlsanity.com/paper/2602.20608/) (2026)
- [HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding](https://mlsanity.com/paper/2603.02329/) (2026)
- [YOLOA: Real-Time Affordance Detection via LLM Adapter](https://mlsanity.com/paper/2512.03418/) (2025)
- [UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation](https://mlsanity.com/paper/2603.23478/) (2026)
- [Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers](https://mlsanity.com/paper/2605.03614/) (2026)
- [AFUN: Towards an Affordance Foundation Model for Functionality Understanding](https://mlsanity.com/paper/2606.02551/) (2026)
- [GROW2: Grounding Which and Where for Robot Tool Use](https://mlsanity.com/paper/2606.30632/) (2026)
- [Zero-shot 2D Grounding with Novel Affordance Types](https://mlsanity.com/paper/2608.08929/) (2026)
- [UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction](https://mlsanity.com/paper/2609.12898/) (2026)
- [UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception](https://mlsanity.com/paper/2609.37264/) (2026)

## Abstract (arXiv)

Identifying affordance regions on 3D objects from semantic cues is essential for robotics and human-machine interaction. However, existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data and a reliance on 3D backbones focused on geometric encoding, which often lack resilience to real-world noise and data corruption. We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models. We employ a dual-branch architecture with Gaussian splatting to establish consistent mappings between 3D point clouds and 2D representations, enabling realistic 2D renderings from sparse point clouds. A granularity-adaptive fusion module and a 2D-3D consistency alignment module further strengthen cross-modal alignment and knowledge transfer, allowing the 3D branch to benefit from the rich semantics and generalization capacity of 2D models. To holistically assess the robustness, we introduce two new corruption-based benchmarks: PIAD-C and LASO-C. Extensive experiments on public datasets and our benchmarks show that GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data, demonstrating robust and adaptable affordance prediction under diverse conditions. Code and corruption datasets have been made publicly available.