mlsanityData to 14 Oct 2026

Egocentric Video-Language Pretraining

arXiv 2206.01670 ↗First posted 3 Jun 2022cs.CV cs.AIcited by 232

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.

Key facts. Cited by 232 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

43 of 60 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2018SlowFast Networks for Video Recognitioncited by 1,867
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  10. 2021Video Swin Transformercited by 1,091
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2017Dense-Captioning Events in Videoscited by 793
31 more
  1. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  2. 2017Localizing Moments in Video with Natural Languagecited by 663
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  4. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  5. 2020Rescaling Egocentric Visioncited by 540
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 285
  10. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  11. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  12. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 235
  13. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  14. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  15. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  16. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  17. 2020Support-set bottlenecks for video-text representation learningcited by 170
  18. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 128
  19. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  20. 2020Video Self-Stitching Graph Network for Temporal Action Localizationcited by 104
  21. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 103
  22. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  23. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  24. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  25. 2018Reconstruction Network for Video Captioningcited by 98
  26. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  27. 2018Localizing Moments in Video with Temporal Languagecited by 83
  28. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  29. 2020VLG-Net: Video-Language Graph Matching Network for Video Groundingcited by 50
  30. 2021On Semantic Similarity in Video Retrievalcited by 46
  31. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 62; 2024: 65; 2025: 58; 2026: 31.

232 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  5. 2022Learning Video Representations from Large Language Modelscited by 205
  6. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 190
  7. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  8. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  9. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  10. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  11. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  12. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
38 more
  1. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  2. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  3. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  4. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 89
  5. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  6. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  7. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  8. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  9. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  10. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  11. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  12. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  13. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  14. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  15. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  16. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  17. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  18. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  19. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  20. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 44
  21. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 44
  22. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 43
  23. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  24. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  25. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 41
  26. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  27. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  28. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  29. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  30. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  31. 2022IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Textcited by 37
  32. 2023Learning Object State Changes in Videos: An Open-World Perspectivecited by 36
  33. 2024SnAG: Scalable and Accurate Video Groundingcited by 36
  34. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 36
  35. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  36. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  37. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  38. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 31

50 most cited of 232 All 232 (JSON)