mlsanityData to 7 Oct 2026

Language Is Not All You Need: Aligning Perception with Language Models

arXiv 2302.14045 ↗First posted 27 Feb 2023cs.CL cs.CVcited by 543

On the map: World models → World models for language agents

Abstract (arXiv)

A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.

Key facts. Cited by 543 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 49 of its 82 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

49 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Language Models are Few-Shot Learners28 May

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
36 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  14. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  20. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  21. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  22. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  23. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  24. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  25. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  26. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  27. 2022A Length-Extrapolatable Transformercited by 123
  28. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  29. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  30. 2022Language Models are General-Purpose Interfacescited by 96
  31. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  32. 2016Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objectscited by 22
  33. 2022Foundation Transformerscited by 19
  34. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  35. 2022Visually-Augmented Language Modelingcited by 18
  36. 2022TorchScale: Transformers at Scalecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 204; 2024: 185; 2025: 98; 2026: 56.

543 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2025Qwen2.5-Omni Technical Reportcited by 704
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  13. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  14. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  15. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  16. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  17. 2024Large Language Models: A Surveycited by 506
  18. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  19. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  22. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  23. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  24. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  25. 2024What matters when building vision-language models?cited by 321
  26. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  27. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  28. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  29. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  30. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  31. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  32. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  33. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  34. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  35. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  36. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  37. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  38. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233

50 most cited of 543 All 543 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Performance of Multimodal Language Models
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  7. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  8. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models