# Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations

arXiv: https://arxiv.org/abs/2503.19706 · HTML: https://mlsanity.com/paper/2503.19706/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 60 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [Actor and Observer: Joint Modeling of First and Third-Person Videos](https://mlsanity.com/paper/1804.09627/) (2018)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2212.04500/) (2022)

## Built on by (most cited first, 6 of 6)

- [IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly](https://mlsanity.com/paper/2604.10409/) (2026)
- [GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding](https://mlsanity.com/paper/2505.10671/) (2025)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency](https://mlsanity.com/paper/2603.09798/) (2026)
- [Revealing Multi-View Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2603.23934/) (2026)
- [V-LynX: Token Interface Alignment for Video+X LLMs](https://mlsanity.com/paper/2606.00508/) (2026)

## Abstract (arXiv)

View-invariant representation learning from egocentric (first-person, ego) and exocentric (third-person, exo) videos is a promising approach toward generalizing video understanding systems across multiple viewpoints. However, this area has been underexplored due to the substantial differences in perspective, motion patterns, and context between ego and exo views. In this paper, we propose a novel masked ego-exo modeling that promotes both causal temporal dynamics and cross-view alignment, called Bootstrap Your Own Views (BYOV), for fine-grained view-invariant video representation learning from unpaired ego-exo videos. We highlight the importance of capturing the compositional nature of human actions as a basis for robust cross-view understanding. Specifically, self-view masking and cross-view masking predictions are designed to learn view-invariant and powerful representations concurrently. Experimental results demonstrate that our BYOV significantly surpasses existing approaches with notable gains across all metrics in four downstream ego-exo video tasks. The code is available at https://github.com/park-jungin/byov.