mlsanityData to 7 Oct 2026

GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition

arXiv 2401.10039 ↗First posted 18 Jan 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs), pre-trained on large-scale datasets, have shown impressive performance in various visual recognition tasks. This advancement paves the way for notable performance in Zero-Shot Egocentric Action Recognition (ZS-EAR). Typically, VLMs handle ZS-EAR as a global video-text matching task, which often leads to suboptimal alignment of vision and linguistic knowledge. We propose a refined approach for ZS-EAR using VLMs, emphasizing fine-grained concept-description alignment that capitalizes on the rich semantic and contextual details in egocentric videos. In this paper, we introduce GPT4Ego, a straightforward yet remarkably potent VLM framework for ZS-EAR, designed to enhance the fine-grained alignment of concept and description between vision and language. Extensive experiments demonstrate GPT4Ego significantly outperforms existing VLMs on three large-scale egocentric video benchmarks, i.e., EPIC-KITCHENS-100 (33.2%, +9.4%), EGTEA (39.6%, +5.5%), and CharadesEgo (31.5%, +2.6%).

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 100 of its 145 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

100 of 145 references.

References · 100

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2023Segment Anythingcited by 8,158
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  14. 2021Video Swin Transformercited by 1,090
  15. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  16. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  17. 2021Multiscale Vision Transformerscited by 896
  18. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  20. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  21. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  22. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  23. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  24. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  25. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  26. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  27. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  28. 2017Temporal Relational Reasoning in Videoscited by 324
  29. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  30. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  31. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  32. 2022Egocentric Video-Language Pretrainingcited by 231
  33. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  34. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  35. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  36. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  37. 2022Learning Video Representations from Large Language Modelscited by 204
  38. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 2.

8 papers.

  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  2. 2024FreeVA: Offline MLLM as Training-Free Video Assistantcited by 29
  3. 2024HOI-Ref: Hand-Object Interaction Referral in Egocentric Visioncited by 9
  4. 2025ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognitioncited by 3
  5. 2024Contrastive Language Video Time Pre-trainingcited by 0
  6. 2024TDS-CLIP: Temporal Difference Side Network for Efficient VideoAction Recognitioncited by 0
  7. 2026A New Multi-Domain Benchmark for Micro-Action Recognition and Detectioncited by 0
  8. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition
  2. 2024X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
  3. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  4. 2026ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
  5. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  6. 2026EgoAVU: Egocentric Audio-Visual Understanding
  7. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approach
  8. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI