mlsanityData to 14 Oct 2026

PaliGemma 2: A Family of Versatile VLMs for Transfer

arXiv 2412.03555 ↗First posted 4 Dec 2024cs.CVcited by 207

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning. The resulting family of base models covering different model sizes and resolutions allows us to investigate factors impacting transfer performance (such as learning rate) and to analyze the interplay between the type of task, model size, and resolution. We further increase the number and breadth of transfer tasks beyond the scope of PaliGemma including different OCR-related tasks such as table structure recognition, molecular structure recognition, music score recognition, as well as long fine-grained captioning and radiography report generation, on which PaliGemma 2 obtains state-of-the-art results.

Key facts. Cited by 207 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 112 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

26 of 112 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  2. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  3. 2024What matters when building vision-language models?3 May
  4. 2024BRAVE: Broadening the visual encoding of vision-language models10 Apr
  5. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
  11. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  12. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
7 more
  1. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  2. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  3. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  5. 2022GriTS: Grid table similarity metric for table structure recognitioncited by 28
  6. 2023An End-to-End Multi-Task Learning Model for Image-based Table Recognitioncited by 14
  7. 2022Pre-training image-language transformers for open-vocabulary taskscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 107; 2026: 98.

207 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  3. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  4. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
  5. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 139
  6. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  7. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  8. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  9. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  10. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  11. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  12. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 47
38 more
  1. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 44
  2. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 42
  3. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  4. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 39
  5. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 38
  6. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  8. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queriescited by 29
  9. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  10. 2025Mellow: a small audio language model for reasoningcited by 27
  11. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 26
  12. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  13. 2025v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoningcited by 25
  14. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  15. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  16. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  17. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 22
  18. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  19. 2025LayerD: Decomposing Raster Graphic Designs into Layerscited by 21
  20. 2025QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Modelscited by 21
  21. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  22. 2025TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Trackingcited by 17
  23. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  24. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 17
  25. 2026FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulationcited by 17
  26. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  27. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
  28. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 14
  29. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 14
  30. 2025GARF: Learning Generalizable 3D Reassembly for Real-World Fracturescited by 13
  31. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 13
  32. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 12
  33. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understandingcited by 11
  34. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 11
  35. 2025Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understandingcited by 11
  36. 2026LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Modelscited by 11
  37. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 10
  38. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10

50 most cited of 207 All 207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding
  2. 2026AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
  3. 2024PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter
  4. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models
  5. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
  6. 2025Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalization
  7. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
  8. 2024Boosting Vision-Language Models with Transduction