mlsanityData to 7 Oct 2026

Chameleon: Mixed-Modal Early-Fusion Foundation Models

arXiv 2405.09818 ↗First posted 16 May 2024cs.CLcited by 904

Not on the map. Why?

Cited by 237 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

Key facts. Cited by 904 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

44 of 61 references.

References · 44

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
32 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023Mistral 7Bcited by 3,065
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  14. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  15. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  16. 2024Mixtral of Expertscited by 1,785
  17. 2020GLU Variants Improve Transformercited by 1,724
  18. 2019Root Mean Square Layer Normalizationcited by 1,425
  19. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  20. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  21. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  22. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  25. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  27. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  28. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  30. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  31. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  32. 2023Pretraining on the Test Set Is All You Needcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 153; 2025: 444; 2026: 307.

904 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  7. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  8. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  9. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  10. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  11. 2024OmniGen: Unified Image Generationcited by 388
  12. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
38 more
  1. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  2. 2024Latent Action Pretraining from Videoscited by 302
  3. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  7. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  8. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  9. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  10. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  11. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  12. 2025Transfer between Modalities with MetaQueriescited by 231
  13. 2025Motus: A Unified Latent Action World Modelcited by 231
  14. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  15. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  16. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  17. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  18. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  19. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  20. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  21. 2024An Introduction to Vision-Language Modelingcited by 175
  22. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  23. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  24. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  25. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  26. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  27. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  28. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  29. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  30. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  31. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  32. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  33. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  34. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  35. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  36. 2024Randomized Autoregressive Visual Generationcited by 125
  37. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  38. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120

50 most cited of 904 All 904 (JSON)