# COBE: Contextualized Object Embeddings from Narrated Instructional Video

arXiv: https://arxiv.org/abs/2007.07306 · HTML: https://mlsanity.com/paper/2007.07306/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 12 of 12)

- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [OneRestore: A Universal Restoration Framework for Composite Degradation](https://mlsanity.com/paper/2407.04621/) (2024)
- [Disentangling Visual Embeddings for Attributes and Objects](https://mlsanity.com/paper/2205.08536/) (2022)
- [Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos](https://mlsanity.com/paper/2203.11637/) (2022)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [Localized Vision-Language Matching for Open-vocabulary Object Detection](https://mlsanity.com/paper/2205.06160/) (2022)
- [SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition](https://mlsanity.com/paper/2204.04796/) (2022)
- [Detours for Navigating Instructional Videos](https://mlsanity.com/paper/2401.01823/) (2024)
- [Detecting Object States vs Detecting Objects: A New Dataset and a Quantitative Experimental Study](https://mlsanity.com/paper/2112.08281/) (2021)
- [On the Efficacy of Text-Based Input Modalities for Action Anticipation](https://mlsanity.com/paper/2401.12972/) (2024)
- [Learning Multiple Object States from Actions via Large Language Models](https://mlsanity.com/paper/2405.01090/) (2024)
- [Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection](https://mlsanity.com/paper/2303.10093/) (2023)

## Abstract (arXiv)

Many objects in the real world undergo dramatic variations in visual appearance. For example, a tomato may be red or green, sliced or chopped, fresh or fried, liquid or solid. Training a single detector to accurately recognize tomatoes in all these different states is challenging. On the other hand, contextual cues (e.g., the presence of a knife, a cutting board, a strainer or a pan) are often strongly indicative of how the object appears in the scene. Recognizing such contextual cues is useful not only to improve the accuracy of object detection or to determine the state of the object, but also to understand its functional properties and to infer ongoing or upcoming human-object interactions. A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms. Instead of relying on manually-labeled data for this task, we propose a new framework for learning Contextualized OBject Embeddings (COBE) from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. This enables the learning of an object representation where concepts relate according to a semantic language metric. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.