# InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges

arXiv: https://arxiv.org/abs/2211.09529 · HTML: https://mlsanity.com/paper/2211.09529/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 60 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)

## Built on by (most cited first, 30 of 48)

- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [VideoLLM: Modeling Video Sequence with Large Language Models](https://mlsanity.com/paper/2305.13292/) (2023)
- [InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language](https://mlsanity.com/paper/2305.05662/) (2023)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos](https://mlsanity.com/paper/2312.05269/) (2023)
- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory](https://mlsanity.com/paper/2301.00746/) (2023)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [PALM: Predicting Actions through Language Models](https://mlsanity.com/paper/2311.17944/) (2023)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2304.03959/) (2023)
- [Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation](https://mlsanity.com/paper/2301.09209/) (2023)
- [GroundNLQ @ Ego4D Natural Language Queries Challenge 2023](https://mlsanity.com/paper/2306.15255/) (2023)
- [Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization](https://mlsanity.com/paper/2211.14053/) (2022)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [SpotEM: Efficient Video Search for Episodic Memory](https://mlsanity.com/paper/2306.15850/) (2023)
- [UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection](https://mlsanity.com/paper/2404.04933/) (2024)
- [EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT](https://mlsanity.com/paper/2510.23569/) (2025)
- [AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2406.01194/) (2024)
- [ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarios](https://mlsanity.com/paper/2309.14809/) (2023)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [Object-Shot Enhanced Grounding Network for Egocentric Video](https://mlsanity.com/paper/2505.04270/) (2025)
- [Boundary-Denoising for Video Activity Localization](https://mlsanity.com/paper/2304.02934/) (2023)
- [ObjectNLQ @ Ego4D Episodic Memory Challenge 2024](https://mlsanity.com/paper/2406.15778/) (2024)
- [OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detection](https://mlsanity.com/paper/2502.20361/) (2025)
- [Graph Propagation Transformer for Graph Representation Learning](https://mlsanity.com/paper/2305.11424/) (2023)

## Abstract (arXiv)

In this report, we present our champion solutions to five tracks at Ego4D challenge. We leverage our developed InternVideo, a video foundation model, for five Ego4D tasks, including Moment Queries, Natural Language Queries, Future Hand Prediction, State Change Object Detection, and Short-term Object Interaction Anticipation. InternVideo-Ego4D is an effective paradigm to adapt the strong foundation model to the downstream ego-centric video understanding tasks with simple head designs. In these five tasks, the performance of InternVideo-Ego4D comprehensively surpasses the baseline methods and the champions of CVPR2022, demonstrating the powerful representation ability of InternVideo as a video foundation model. Our code will be released at https://github.com/OpenGVLab/ego4d-eccv2022-solutions