# Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022

arXiv: https://arxiv.org/abs/2206.07689 · HTML: https://mlsanity.com/paper/2206.07689/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 10 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)

## Built on by (most cited first, 3 of 3)

- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [Egocentric Video Task Translation @ Ego4D Challenge 2022](https://mlsanity.com/paper/2302.01891/) (2023)
- [Masked Autoencoders for Egocentric Video Understanding @ Ego4D Challenge 2022](https://mlsanity.com/paper/2211.15286/) (2022)

## Abstract (arXiv)

This technical report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of object tokens that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a "Frame-Clip Consistency" loss, which ensures the flow of structured information between images and videos. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error.