mlsanityData to 14 Oct 2026

Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

arXiv 2402.07865 ↗First posted 12 Feb 2024cs.CV cs.AI cs.CLcited by 339

Not on the map. Why?

Cited by 171 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance - a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs.

Key facts. Cited by 339 papers in the map (data through 14 Oct 2026). Builds on 60 of its 75 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

60 of 75 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  13. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  14. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  15. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  16. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  17. 2019Towards VQA Models That Can Readcited by 1,718
  18. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  19. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  20. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  21. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  22. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  23. 2016Modeling Context in Referring Expressionscited by 1,063
  24. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  25. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  26. 2016A Diagram Is Worth A Dozen Imagescited by 804
  27. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  28. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  29. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  30. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  31. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  32. 2019Defending Against Neural Fake Newscited by 559
  33. 2023LERF: Language Embedded Radiance Fieldscited by 543
  34. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  35. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  36. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  37. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  38. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 72; 2025: 143; 2026: 124.

339 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  6. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  7. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  8. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  9. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  11. 2024What matters when building vision-language models?cited by 326
  12. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
38 more
  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  3. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  4. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 192
  5. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  6. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  7. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  8. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
  9. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  10. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  11. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  12. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  13. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  14. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  15. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  16. 2025MiMo-VL Technical Reportcited by 115
  17. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  18. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  19. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  20. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
  21. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 105
  22. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  23. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 93
  24. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  25. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  26. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  27. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  28. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  29. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  30. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  31. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 80
  32. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 80
  33. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  34. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 76
  35. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  36. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  37. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 73
  38. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73

50 most cited of 339 All 339 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  3. 2024Probing Visual Language Priors in VLMs
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024An Introduction to Vision-Language Modeling
  6. 2025Visual Large Language Models for Generalized and Specialized Applications
  7. 2025Hidden in plain sight: VLMs overlook their visual representations
  8. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages