mlsanityData to 7 Oct 2026

Learning Video Representations from Large Language Models

arXiv 2212.04501 ↗First posted 8 Dec 2022cs.CVcited by 204

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs). We repurpose pre-trained LLMs to be conditioned on visual input, and finetune them to create automatic video narrators. Our auto-generated narrations offer a number of advantages, including dense coverage of long videos, better temporal synchronization of the visual information and text, and much higher diversity of text. The video-text embedding learned contrastively with these additional auto-generated narrations outperforms the previous state-of-the-art on multiple first-person and third-person video tasks, both in zero-shot and finetuned setups. Most notably, LaViLa obtains an absolute gain of 10.1% on EGTEA classification and 5.9% Epic-Kitchens-100 multi-instance retrieval benchmarks. Furthermore, LaViLa trained with only half the narrations from the Ego4D dataset outperforms baseline models trained on the full set, and shows positive scaling behavior on increasing pre-training data and model size.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Builds on 69 of its 92 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

69 of 92 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2016Layer Normalizationcited by 5,579
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2021Scaling Vision Transformerscited by 993
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  15. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  16. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  17. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  19. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  20. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  21. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  22. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  23. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  24. 2020Rescaling Egocentric Visioncited by 529
  25. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  26. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  27. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  28. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  29. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  30. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  31. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  32. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  33. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  34. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  35. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  36. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  37. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  38. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 44; 2024: 75; 2025: 58; 2026: 27.

204 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  6. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  9. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  10. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  11. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  12. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
38 more
  1. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  2. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  3. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  4. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  5. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  6. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  7. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  8. 2023An Outlook into the Future of Egocentric Visioncited by 84
  9. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  10. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  11. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  12. 2024Language Repository for Long Video Understandingcited by 55
  13. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 53
  14. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  15. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  16. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  17. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  18. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  19. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  20. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  21. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  22. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  23. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  24. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  25. 2023Learning Object State Changes in Videos: An Open-World Perspectivecited by 36
  26. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  27. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  28. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  29. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  30. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  31. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  32. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  33. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  34. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  35. 2024Universal Time-Series Representation Learning: A Surveycited by 28
  36. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  37. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  38. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24

50 most cited of 204 All 204 (JSON)