mlsanityData to 7 Oct 2026

ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation

arXiv 2407.06135 ↗First posted 8 Jul 2024cs.CL cs.AI cs.CVcited by 112

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Previous open-source large multimodal models (LMMs) have faced several limitations: (1) they often lack native integration, requiring adapters to align visual representations with pre-trained large language models (LLMs); (2) many are restricted to single-modal generation; (3) while some support multimodal generation, they rely on separate diffusion models for visual modeling and generation. To mitigate these limitations, we present Anole, an open, autoregressive, native large multimodal model for interleaved image-text generation. We build Anole from Meta AI's Chameleon, adopting an innovative fine-tuning strategy that is both data-efficient and parameter-efficient. Anole demonstrates high-quality, coherent multimodal generation capabilities. We have open-sourced our model, training framework, and instruction tuning data.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 29 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

23 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  6. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep

Other references · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2023Textbooks Are All You Needcited by 561
  11. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  12. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
5 more
  1. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  2. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  3. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  4. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  5. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 55; 2026: 42.

112 papers.

Sort:

By area:

  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2024In-Context LoRA for Diffusion Transformerscited by 155
  4. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  5. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  8. 2025Visual Planning: Let's Think Only with Imagescited by 85
  9. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  10. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  11. 2025Thinking with Generated Imagescited by 71
  12. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
38 more
  1. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  2. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  3. 2024Interleaved-Modal Chain-of-Thoughtcited by 47
  4. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  5. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  6. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  7. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  8. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  9. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  10. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  11. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  12. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  13. 2024CAR: Controllable Autoregressive Modeling for Visual Generationcited by 29
  14. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  15. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  16. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
  17. 2025YoChameleon: Personalized Vision and Language Generationcited by 20
  18. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  19. 2024Multimodal Markup Document Models for Graphic Design Completioncited by 15
  20. 2025Watermarking Autoregressive Image Generationcited by 15
  21. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generationcited by 15
  22. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 15
  23. 2025LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Modelscited by 11
  24. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 11
  25. 2025Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropycited by 11
  26. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  27. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  28. 2026Language-Conditioned World Modeling for Visual Navigationcited by 9
  29. 2024IDEA-Bench: How Far are Generative Models from Professional Designing?cited by 7
  30. 2025UniEdit-I: Training-free Image Editing for Unified VLM via Iterative Understanding, Editing and Verifyingcited by 7
  31. 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoningcited by 7
  32. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 7
  33. 2024MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistantscited by 6
  34. 2025Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generationcited by 6
  35. 2025VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generationcited by 6
  36. 2025CSD-VAR: Content-Style Decomposition in Visual Autoregressive Modelscited by 6
  37. 2025The Promise of RL for Autoregressive Image Editingcited by 6
  38. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation
  4. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
  5. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  6. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  7. 2023Generating Images with Multimodal Language Models
  8. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens