mlsanityData to 14 Oct 2026

OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

arXiv 2308.01390 ↗First posted 2 Aug 2023cs.CV cs.AI cs.LGcited by 565

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters. OpenFlamingo is an ongoing effort to produce an open-source replication of DeepMind's Flamingo models. On seven vision-language datasets, OpenFlamingo models average between 80 - 89% of corresponding Flamingo performance. This technical report describes our models, training data, hyperparameters, and evaluation suite. We share our models and code at https://github.com/mlfoundations/open_flamingo.

Key facts. Cited by 565 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

38 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
24 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  13. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  16. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  17. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  18. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  19. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  20. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  21. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  22. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  23. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 56
  24. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 76; 2024: 249; 2025: 171; 2026: 69.

565 papers.

Sort:

By area:

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2024Long Context Transfer from Language to Visioncited by 515
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  12. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
38 more
  1. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  3. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  4. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  5. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  6. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  8. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  9. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  10. 2024What matters when building vision-language models?cited by 326
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  12. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  13. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  14. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  15. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  16. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  17. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  18. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  19. 2023Video Understanding with Large Language Models: A Surveycited by 206
  20. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  21. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  22. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  23. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  24. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  25. 2024An Introduction to Vision-Language Modelingcited by 178
  26. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  27. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  28. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  29. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  30. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  31. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  32. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  33. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  34. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  35. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  36. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  37. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  38. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129

50 most cited of 565 All 565 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  2. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese
  3. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  4. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies
  5. 2026Multilingual Training and Evaluation Resources for Vision-Language Models
  6. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
  7. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
  8. 2021FLAVA: A Foundational Language And Vision Alignment Model