mlsanityData to 7 Oct 2026

Gemma 3 Technical Report

arXiv 2503.19786 ↗First posted 25 Mar 2025cs.CL cs.AIcited by 1,686

Not on the map. Why?

Cited by 435 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.

Key facts. Cited by 1,686 papers in the map (data through 7 Oct 2026). Builds on 79 of its 92 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

79 of 92 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  15. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  19. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  20. 2023PaLM 2 Technical Reportcited by 1,154
  21. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  22. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  23. 2021Ethical and social risks of harm from Language Modelscited by 957
  24. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  25. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  26. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  27. 2016A Diagram Is Worth A Dozen Imagescited by 793
  28. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  29. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  30. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  31. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  32. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  33. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  34. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  35. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  36. 2021InfographicVQAcited by 430
  37. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  38. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 711; 2026: 975.

1,686 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  4. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  5. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  7. 2025Generalizing Verifiable Instruction Followingcited by 178
  8. 2026MiMo-V2-Flash Technical Reportcited by 163
  9. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  10. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  11. 2025Training Agents Inside of Scalable World Modelscited by 122
  12. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
38 more
  1. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  2. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  3. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  4. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  6. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  7. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  8. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  9. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
  10. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
  11. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  12. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  13. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  14. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  15. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  16. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 52
  17. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  18. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  19. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  20. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  21. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  22. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  23. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 45
  24. 2025SensorLM: Learning the Language of Wearable Sensorscited by 44
  25. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  26. 2025Text-to-CadQuery: A New Paradigm for CAD Generation with Scalable Large Model Capabilitiescited by 43
  27. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  28. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  29. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  30. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  31. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  32. 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 36
  33. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  34. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
  35. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  36. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  37. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 32
  38. 2025BitNet b1.58 2B4T Technical Reportcited by 30

50 most cited of 1,686 All 1,686 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Gemma 4 Technical Report
  2. 2024PALO: A Polyglot Large Multimodal Model for 5B People
  3. 2026TranslateGemma Technical Report
  4. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  5. 2025MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding
  6. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  7. 2023On the Performance of Multimodal Language Models
  8. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning