mlsanityData to 7 Oct 2026

Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

arXiv 2311.08046 ↗First posted 14 Nov 2023cs.CVcited by 368

Not on the map. Why?

Cited by 164 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multimodal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conversations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to efficiently utilize a limited number of visual tokens to simultaneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level semantic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Extensive experimental results demonstrate that Chat-UniVi consistently outperforms even existing methods exclusively designed for either images or videos. Code is available at https://github.com/PKU-YuanGroup/Chat-UniVi.

Key facts. Cited by 368 papers in the map (data through 7 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

66 of 86 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  15. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  17. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  18. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  19. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  20. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  21. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  22. 2022Token Merging: Your ViT But Fastercited by 995
  23. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  24. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  25. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  26. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  27. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  28. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  29. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  30. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  31. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  32. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  33. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  34. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  35. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  36. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  37. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  38. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 123; 2025: 155; 2026: 85.

368 papers.

Sort:

By area:

  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024Long Context Transfer from Language to Visioncited by 502
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  9. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  10. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  11. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  12. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
38 more
  1. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  2. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  3. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  7. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2023Video Understanding with Large Language Models: A Surveycited by 203
  10. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  11. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  12. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  13. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  14. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  15. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  16. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  17. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  18. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  19. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  20. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  21. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  22. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  23. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  24. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  25. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  26. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  27. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  28. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  29. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  30. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  31. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  32. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  33. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  34. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  35. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  36. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  37. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  38. 2025Scaling RL to Long Videoscited by 83

50 most cited of 368 All 368 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  2. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  4. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  5. 2026Chatting with Images for Introspective Visual Thinking
  6. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  7. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  8. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer