mlsanityData to 7 Oct 2026

CogVLM: Visual Expert for Pretrained Language Models

arXiv 2311.03079 ↗First posted 6 Nov 2023cs.CVcited by 704

Not on the map. Why?

Cited by 406 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X 55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.

Key facts. Cited by 704 papers in the map (data through 7 Oct 2026). Builds on 56 of its 74 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

56 of 74 references.

References · 56

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  15. 2016Modeling Context in Referring Expressionscited by 1,054
  16. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  17. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  18. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  22. 2021GPT Understands, Toocited by 715
  23. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  24. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  25. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  26. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  27. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  28. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  31. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  32. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  33. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  34. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  35. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  36. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  37. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  38. 2023Emu: Generative Pretraining in Multimodalitycited by 156

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 28; 2024: 341; 2025: 258; 2026: 77.

704 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
38 more
  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  6. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  7. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  11. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  13. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  14. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  16. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  17. 2024What matters when building vision-language models?cited by 321
  18. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  19. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  20. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  21. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  22. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  23. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  24. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  25. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  26. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  27. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  28. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  29. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  30. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  31. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  32. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  33. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  34. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  35. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  36. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  37. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  38. 2024Building and better understanding vision-language models: insights and future directionscited by 169

50 most cited of 704 All 704 (JSON)