mlsanityData to 14 Oct 2026

Retrieval-Augmented Egocentric Video Captioning

arXiv 2401.00789 ↗First posted 1 Jan 2024cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

Understanding human actions from videos of first-person view poses significant challenges. Most prior approaches explore representation learning on egocentric videos only, while overlooking the potential benefit of exploiting existing large-scale third-person videos. In this paper, (1) we develop EgoInstructor, a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos to enhance the video captioning of egocentric videos. (2) For training the cross-view retrieval module, we devise an automatic pipeline to discover ego-exo video pairs from distinct large-scale egocentric and exocentric datasets. (3) We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions. (4) Through extensive experiments, our cross-view retrieval module demonstrates superior performance across seven benchmarks. Regarding egocentric video captioning, EgoInstructor exhibits significant improvements by leveraging third-person videos as references. Project page is available at: https://jazzcharles.github.io/Egoinstructor/

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 64 of its 85 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

64 of 85 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  10. 2020Rescaling Egocentric Visioncited by 540
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  12. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  13. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  14. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  15. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  16. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  17. 2022Egocentric Video-Language Pretrainingcited by 232
  18. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  19. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  20. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  21. 2022Learning Video Representations from Large Language Modelscited by 205
  22. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  23. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  24. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  25. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 103
  26. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  27. 2022Temporal Alignment Networks for Long-term Videocited by 84
  28. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  29. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  30. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  31. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  32. 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
  33. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 53
  34. 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
  35. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  36. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 47
  37. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  38. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 22; 2026: 5.

42 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  2. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  3. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  4. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  5. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  6. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  7. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  8. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  9. 2024EgoLM: Multi-Modal Language Model of Egocentric Motionscited by 28
  10. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 23
  11. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 21
  12. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
30 more
  1. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  2. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  3. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  4. 2024Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognitioncited by 18
  5. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  6. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  7. 2024R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generationcited by 12
  8. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  9. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 8
  10. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 8
  11. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  12. 2025SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understandingcited by 7
  13. 2024Egocentric and Exocentric Methods: A Short Surveycited by 6
  14. 2024TechCoach: Towards Technical-Point-Aware Descriptive Action Coachingcited by 6
  15. 2025Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMscited by 6
  16. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  17. 2024Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videoscited by 4
  18. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  19. 2025EgoPrivacy: What Your First-Person Camera Says About You?cited by 4
  20. 2025DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Descriptioncited by 3
  21. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 3
  22. 2025Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidancecited by 2
  23. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  24. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  25. 2026Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistencycited by 2
  26. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1
  27. 2025MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understandingcited by 1
  28. 2026MARS: Technical Report for the CASTLE Challenge at EgoVis 2026cited by 1
  29. 2025The N-Body Problem: Parallel Execution from Single-Person Egocentric Videocited by 0
  30. 2026Learning to Rank Caption Chains for Video-Text Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
  2. 2026LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
  3. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  4. 2026EgoAVU: Egocentric Audio-Visual Understanding
  5. 2026CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
  6. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding
  7. 2024EAGLE: Egocentric AGgregated Language-video Engine
  8. 2026EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory