# Anticipating Visual Representations from Unlabeled Video

arXiv: https://arxiv.org/abs/1504.08023 · HTML: https://mlsanity.com/paper/1504.08023/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2015.

## Built on by (most cited first, 14 of 14)

- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Playable Video Generation](https://mlsanity.com/paper/2101.12195/) (2021)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [Scaling 4D Representations](https://mlsanity.com/paper/2412.15212/) (2024)
- [Playable Environments: Video Manipulation in Space and Time](https://mlsanity.com/paper/2203.01914/) (2022)
- [3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes](https://mlsanity.com/paper/2304.11470/) (2023)
- [CoCon: Cooperative-Contrastive Learning](https://mlsanity.com/paper/2104.14764/) (2021)
- [Click to Move: Controlling Video Generation with Sparse Motion](https://mlsanity.com/paper/2108.08815/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion](https://mlsanity.com/paper/2101.10777/) (2021)
- [IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2311.15193/) (2023)
- [Learning to Anticipate Future with Dynamic Context Removal](https://mlsanity.com/paper/2204.02587/) (2022)
- [Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos](https://mlsanity.com/paper/2204.13548/) (2022)

## Abstract (arXiv)

Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.