mlsanityData to 7 Oct 2026

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

arXiv 2402.03161 ↗First posted 5 Feb 2024cs.CV cs.CLcited by 98

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In light of recent advances in multimodal Large Language Models (LLMs), there is increasing attention to scaling them from image-text data to more informative real-world videos. Compared to static images, video poses unique challenges for effective large-scale pre-training due to the modeling of its spatiotemporal dynamics. In this paper, we address such limitations in video-language pre-training with an efficient video decomposition that represents each video as keyframes and temporal motions. These are then adapted to an LLM using well-designed tokenizers that discretize visual and temporal information as a few tokens, thus enabling unified generative pre-training of videos, images, and text. At inference, the generated tokens from the LLM are carefully recovered to the original continuous pixel space to create various video content. Our proposed framework is both capable of comprehending and generating image and video content, as demonstrated by its competitive performance across 13 multimodal benchmarks in image and video understanding and generation. Our code and models are available at https://video-lavit.github.io.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 98 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

82 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  13. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2022Video Diffusion Modelscited by 2,280
  16. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  17. 2017The Kinetics Human Action Video Datasetcited by 1,894
  18. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  19. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  20. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  21. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  22. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  23. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  24. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  25. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  26. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  27. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  28. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  29. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  30. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  31. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  32. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  33. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  34. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  35. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  36. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  37. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  38. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 46; 2026: 24.

98 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  4. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  5. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  6. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  7. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  8. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  9. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  10. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  11. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  12. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
38 more
  1. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  2. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  3. 2024Foundation Models for Video Understanding: A Surveycited by 46
  4. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  5. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  6. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  7. 2024Autoregressive Models in Vision: A Surveycited by 39
  8. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  9. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  10. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  11. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  12. 2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 30
  13. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  14. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  15. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  16. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  17. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  18. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  19. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 21
  20. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  21. 2025MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understandingcited by 16
  22. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  23. 2024Personalized Image Generation with Large Multimodal Modelscited by 14
  24. 2024VidTwin: Video VAE with Decoupled Structure and Dynamicscited by 14
  25. 2025Efficient Motion-Aware Video MLLMcited by 14
  26. 2025HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generationcited by 13
  27. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  28. 2024DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframescited by 12
  29. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  30. 2024SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretizationcited by 12
  31. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  32. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  33. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  34. 2025Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusioncited by 10
  35. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  36. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  37. 2025MotionStrata: Hierarchical Motion Latents for Compact Video Autoencodingcited by 9
  38. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
  2. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  3. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
  4. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspective
  5. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  6. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understanding
  7. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
  8. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding