# Informative Text-Image Alignment for Visual Affordance Learning with Foundation Models

arXiv: https://arxiv.org/abs/2509.17074 · HTML: https://mlsanity.com/paper/2509.17074/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Learning to Act Properly: Predicting and Explaining Affordances from Images](https://mlsanity.com/paper/1712.07576/) (2017)
- [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [Bayesian Deep Learning for Affordance Segmentation in images](https://mlsanity.com/paper/2303.00871/) (2023)
- [STRAP: Structured Object Affordance Segmentation with Point Supervision](https://mlsanity.com/paper/2304.08492/) (2023)
- [Learning Instruction-Guided Manipulation Affordance via Large Models for Embodied Robotic Tasks](https://mlsanity.com/paper/2408.10658/) (2024)

## Abstract (arXiv)

Visual affordance learning is crucial for robots to understand and interact effectively with the physical world. Recent advances in this field attempt to leverage pre-trained knowledge of vision-language foundation models to learn affordance properties with limited training data, providing a novel paradigm for visual affordance learning. However, these methods overlook the significance of maintaining feature alignment between visual images and language descriptions for identifying affordance areas with textual guidance, and thus may lead to suboptimal results. In this paper, we present an informative framework for text-guided affordance learning, which involves information-based constraints to achieve text-image alignment at feature level. Specifically, we design an affordance mutual information constraint that helps learn appropriate textual prompts and task-oriented visual features simultaneously by maximizing the mutual information between the features of the affordance areas in the input images and the corresponding textual prompts. In addition, we propose an object-level information constraint that maximizes the mutual information between the visual features of a given object and the text features of the category it belongs to. This enables the model to capture high-quality representations for the object, providing more reliable semantic priors for identifying affordance regions. Experimental results on the AGD20K dataset show that the proposed method outperforms existing approaches and achieves the new state-of-the-art in one-shot affordance learning.