# Anticipating Object State Changes in Long Procedural Videos

arXiv: https://arxiv.org/abs/2405.12789 · HTML: https://mlsanity.com/paper/2405.12789/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 30 of 35)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [Next-Active-Object prediction from Egocentric Videos](https://mlsanity.com/paper/1904.05250/) (2019)
- [Disentangling Visual Embeddings for Attributes and Objects](https://mlsanity.com/paper/2205.08536/) (2022)
- [Breaking the "Object" in Video Object Segmentation](https://mlsanity.com/paper/2212.06200/) (2022)
- [InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges](https://mlsanity.com/paper/2211.09529/) (2022)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos](https://mlsanity.com/paper/2312.07322/) (2023)
- [Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos](https://mlsanity.com/paper/2203.11637/) (2022)
- [Attend and Interact: Higher-Order Object Interactions for Video Understanding](https://mlsanity.com/paper/1711.06330/) (2017)
- [Interaction Region Visual Transformer for Egocentric Action Anticipation](https://mlsanity.com/paper/2211.14154/) (2022)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [Actions ~ Transformations](https://mlsanity.com/paper/1512.00795/) (2015)
- [Joint Discovery of Object States and Manipulation Actions](https://mlsanity.com/paper/1702.02738/) (2017)
- [Object-centric Video Representation for Long-term Action Anticipation](https://mlsanity.com/paper/2311.00180/) (2023)
- [StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2304.03959/) (2023)
- [Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation](https://mlsanity.com/paper/2301.09209/) (2023)
- [How can objects help action recognition?](https://mlsanity.com/paper/2306.11726/) (2023)
- [Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videos](https://mlsanity.com/paper/2308.08303/) (2023)
- [Chop & Learn: Recognizing and Generating Object-State Compositions](https://mlsanity.com/paper/2309.14339/) (2023)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)

## Built on by (most cited first, 2 of 2)

- [IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Construction](https://mlsanity.com/paper/2605.01666/) (2026)
- [Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes](https://mlsanity.com/paper/2608.04426/) (2026)

## Abstract (arXiv)

In this work, we introduce (a) the new problem of anticipating object state changes in images and videos during procedural activities, (b) new curated annotation data for object state change classification based on the Ego4D dataset, and (c) the first method for addressing this challenging problem. Solutions to this new task have important implications in vision-based scene understanding, automated monitoring systems, and action planning. The proposed novel framework predicts object state changes that will occur in the near future due to yet unseen human actions by integrating learned visual features that represent recent visual information with natural language (NLP) features that represent past object state changes and actions. Leveraging the extensive and challenging Ego4D dataset which provides a large-scale collection of first-person perspective videos across numerous interaction scenarios, we introduce an extension noted Ego4D-OSCA that provides new curated annotation data for the object state change anticipation task (OSCA). An extensive experimental evaluation is presented demonstrating the proposed method's efficacy in predicting object state changes in dynamic scenarios. The performance of the proposed approach also underscores the potential of integrating video and linguistic cues to enhance the predictive performance of video understanding systems and lays the groundwork for future research on the new task of object state change anticipation. The source code and the new annotation data (Ego4D-OSCA) will be made publicly available.