mlsanityData to 7 Oct 2026

CogVLM2: Visual Language Models for Image and Video Understanding

arXiv 2408.16500 ↗First posted 29 Aug 2024cs.CVcited by 214

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to 1344 × 1344 pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.

Key facts. Cited by 214 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 76 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

68 of 76 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text10 Jun
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  3. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  4. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM27 Mar
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 63

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
38 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  13. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  14. 2023A Survey on Multimodal Large Language Modelscited by 898
  15. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  18. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  19. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  20. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  21. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  22. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  23. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  24. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  25. 2021InfographicVQAcited by 430
  26. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  27. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  28. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  29. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  30. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  31. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  32. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  33. 2019Scene Text Visual Question Answeringcited by 327
  34. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  35. 2022Visual Spatial Reasoningcited by 288
  36. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  37. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  38. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 133; 2026: 41.

214 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  6. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  7. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  10. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  11. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  12. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
38 more
  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  2. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  3. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  4. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  5. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  6. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  7. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  8. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  9. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  10. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMscited by 46
  11. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  12. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  13. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 42
  14. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  15. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  16. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  17. 2024MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3Dcited by 34
  18. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
  19. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  20. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  21. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  22. 2025EliGen: Entity-Level Controlled Image Generation with Regional Attentioncited by 32
  23. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  24. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
  25. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  26. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  27. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  28. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  29. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  30. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  31. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  32. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  33. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  34. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 23
  35. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  36. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  37. 2025FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Videocited by 21
  38. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 21

50 most cited of 214 All 214 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  2. 2024An Introduction to Vision-Language Modeling
  3. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
  4. 2025Visual Large Language Models for Generalized and Specialized Applications
  5. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  7. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  8. 2025Event-Priori-Based Vision-Language Model for Efficient Visual Understanding