mlsanityData to 7 Oct 2026

EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone

arXiv 2307.05463 ↗First posted 11 Jul 2023cs.CVcited by 130

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks. However, existing egocentric VLP frameworks utilize separate video and language encoders and learn task-specific cross-modal information only during fine-tuning, limiting the development of a unified system. In this work, we introduce the second generation of egocentric video-language pre-training (EgoVLPv2), a significant improvement from the previous generation, by incorporating cross-modal fusion directly into the video and language backbones. EgoVLPv2 learns strong video-text representation during pre-training and reuses the cross-modal attention modules to support different downstream tasks in a flexible and efficient manner, reducing fine-tuning costs. Moreover, our proposed fusion in the backbone strategy is more lightweight and compute-efficient than stacking additional fusion-specific layers. Extensive experiments on a wide range of VL tasks demonstrate the effectiveness of EgoVLPv2 by achieving consistent state-of-the-art performance over strong baselines across all downstream. Our project page can be found at https://shramanpramanick.github.io/EgoVLPv2/.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 97 of its 132 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

97 of 132 references.

References · 97

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2018SlowFast Networks for Video Recognitioncited by 1,862
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2021Video Swin Transformercited by 1,090
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  13. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  14. 2021Multiscale Vision Transformerscited by 896
  15. 2017Dense-Captioning Events in Videoscited by 789
  16. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  17. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  18. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  19. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  20. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  22. 2020Rescaling Egocentric Visioncited by 529
  23. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  24. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  25. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  26. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  27. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  28. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  29. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  30. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  31. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  32. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  33. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  34. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  35. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  36. 2022Egocentric Video-Language Pretrainingcited by 231
  37. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  38. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 39; 2025: 56; 2026: 27.

130 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2023Video Understanding with Large Language Models: A Surveycited by 203
  3. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  4. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  5. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  6. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  7. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  8. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  9. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  10. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  12. 2023PALM: Predicting Actions through Language Modelscited by 28
38 more
  1. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  2. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  3. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  4. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
  5. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  6. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  7. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  8. 2024EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrievalcited by 21
  9. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  10. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  11. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  12. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 19
  13. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answeringcited by 19
  14. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  15. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  16. 2025BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimationcited by 17
  17. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  18. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  19. 2024MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Versioncited by 14
  20. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  21. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  22. 2023Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videoscited by 11
  23. 2024Step Differences in Instructional Videocited by 11
  24. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  25. 2025Object-Shot Enhanced Grounding Network for Egocentric Videocited by 11
  26. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  27. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  28. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 11
  29. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligencecited by 10
  30. 2024A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectivescited by 9
  31. 2024HOI-Ref: Hand-Object Interaction Referral in Egocentric Visioncited by 9
  32. 2024Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Cameracited by 9
  33. 2024ViLCo-Bench: VIdeo Language COntinual learning Benchmarkcited by 9
  34. 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 9
  35. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  36. 2024Every Shot Counts: Using Exemplars for Repetition Counting in Videoscited by 8
  37. 2024FIction: 4D Future Interaction Prediction from Videocited by 8
  38. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8

50 most cited of 130 All 130 (JSON)