# Egocentric Instruction-oriented Affordance Prediction via Large Multimodal Model

arXiv: https://arxiv.org/abs/2508.17922 · HTML: https://mlsanity.com/paper/2508.17922/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 1 of its 59 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 1 of 1)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)

## Built on by (most cited first, 1 of 1)

- [EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting](https://mlsanity.com/paper/2605.07642/) (2026)

## Abstract (arXiv)

Affordance is crucial for intelligent robots in the context of object manipulation. In this paper, we argue that affordance should be task-/instruction-dependent, which is overlooked by many previous works. That is, different instructions can lead to different manipulation regions and directions even for the same object. According to this observation, we present a new dataset comprising fifteen thousand object-instruction-affordance triplets. All scenes in the dataset are from an egocentric viewpoint, designed to approximate the perspective of a human-like robot. Furthermore, we investigate how to enable large multimodal models (LMMs) to serve as affordance predictors by implementing a ``search against verifiers'' pipeline. An LMM is asked to progressively predict affordances, with the output at each step being verified by itself during the iterative process, imitating a reasoning process. Experiments show that our method not only unlocks new instruction-oriented affordance prediction capabilities, but also achieves outstanding performance broadly.