mlsanityData to 14 Oct 2026

Kosmos-2: Grounding Multimodal Large Language Models to the World

arXiv 2306.14824 ↗First posted 26 Jun 2023cs.CL cs.CVcited by 1,089

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.

Key facts. Cited by 1,089 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 68 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

50 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions24 May
  2. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  3. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models23 May
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
31 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2021Grounded Language-Image Pre-trainingcited by 1,156
  7. 2016Modeling Context in Referring Expressionscited by 1,063
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  14. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  18. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  19. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  20. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  21. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  22. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  23. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  24. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  25. 2022Language Models are General-Purpose Interfacescited by 97
  26. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  27. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  28. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  29. 2022Foundation Transformerscited by 19
  30. 2022TorchScale: Transformers at Scalecited by 10
  31. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 139; 2024: 397; 2025: 322; 2026: 231.

1,089 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,344
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
38 more
  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  4. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  12. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  13. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  14. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  15. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  16. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  17. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  18. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  19. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  20. 2024OmniGen: Unified Image Generationcited by 394
  21. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  22. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  23. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  24. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  25. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  26. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  27. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  28. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  29. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  30. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  31. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  32. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  33. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  34. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  35. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  36. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  37. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  38. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279

50 most cited of 1,089 All 1,089 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  2. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  3. 2023Resolving References in Visually-Grounded Dialogue via Text Generation
  4. 2025Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
  5. 2025KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge
  6. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
  7. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
  8. 2024ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension