mlsanityData to 7 Oct 2026

PaLI-3 Vision Language Models: Smaller, Faster, Stronger

arXiv 2310.09199 ↗First posted 13 Oct 2023cs.CVcited by 125

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 61 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

55 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
38 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2021Scaling Vision Transformerscited by 993
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2017Dense-Captioning Events in Videoscited by 789
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  14. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  15. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  17. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  18. 2021InfographicVQAcited by 430
  19. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  20. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  21. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  22. 2020Are we done with ImageNet?cited by 329
  23. 2019Scene Text Visual Question Answeringcited by 327
  24. 2022UL2: Unifying Language Learning Paradigmscited by 292
  25. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  26. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  27. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  28. 2018TallyQA: Answering Complex Counting Questionscited by 152
  29. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  30. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  31. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  32. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  33. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  34. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  35. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  36. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  37. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  38. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 66; 2025: 35; 2026: 12.

125 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  5. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  6. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  7. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  8. 2024What matters when building vision-language models?cited by 321
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  10. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 169
38 more
  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  2. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  3. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  6. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  7. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  8. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  9. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  10. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  11. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  12. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  13. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  14. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  15. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  16. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  17. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  18. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  19. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  20. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  21. 2024Foundation Models for Video Understanding: A Surveycited by 46
  22. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  23. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  24. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  25. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  26. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  27. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  28. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  29. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  30. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  31. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  32. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  33. 2024Scaling 4D Representationscited by 30
  34. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  35. 2024M&M VTO: Multi-Garment Virtual Try-On and Editingcited by 29
  36. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  37. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  38. 2024DocVLM: Make Your VLM an Efficient Readercited by 27

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2026An overview of 3D Vision-Language Models
  4. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  7. 2022A Survey of Vision-Language Pre-Trained Models
  8. 2025jina-vlm: Small Multilingual Vision Language Model