mlsanityData to 7 Oct 2026

Gemma 4 Technical Report

arXiv 2607.02770 ↗First posted 2 Jul 2026cs.CL cs.AIcited by 187

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 50 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
30 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  5. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  6. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  7. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  9. 2025Humanity's Last Examcited by 467
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  12. 2021InfographicVQAcited by 430
  13. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  14. 2020Query-Key Normalization for Transformerscited by 266
  15. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  16. 2024Rotary Position Embedding for Vision Transformercited by 226
  17. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  18. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  19. 2025Generalizing Verifiable Instruction Followingcited by 178
  20. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  21. 2026MiMo-V2-Flash Technical Reportcited by 163
  22. 2026Qwen3.5-Omni Technical Reportcited by 146
  23. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  24. 2025BIG-Bench Extra Hardcited by 123
  25. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  26. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  27. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  28. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  29. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  30. 2026Do Transformers Need Three Projections? Systematic Study of QKV Variantscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 187.

187 papers.

Sort:

By area:

  1. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  2. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  3. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 9
  4. 2026On-Policy Delta Distillationcited by 8
  5. 2026Evo-Bench: Can Language Models Improve Agent Harness?cited by 8
  6. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  7. 2026WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolutioncited by 7
  8. 2026Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memorycited by 7
  9. 2026Metis: Memory Foundation Modelcited by 5
  10. 2026MOSS-VL Technical Reportcited by 5
  11. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 4
  12. 2026JevOut: Natural Context Can Flip Decision Modelscited by 4
38 more
  1. 2026ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillationcited by 3
  2. 2026Extremely Sparse Supervision Incentivizes Reasoning Abilitycited by 3
  3. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 3
  4. 2026A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forevercited by 2
  5. 2026Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleetscited by 2
  6. 2026PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inferencecited by 1
  7. 2026AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approachcited by 1
  8. 2026Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Modelscited by 1
  9. 2026MESA:Task-Adaptive Multi-Structure Evidence Selection for Long-Horizon Agent Memorycited by 1
  10. 2026Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequencescited by 1
  11. 2026Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Gamescited by 1
  12. 2026QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstructioncited by 1
  13. 2026Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Rankingcited by 1
  14. 2026Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agentscited by 1
  15. 2026DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmarkcited by 1
  16. 2026Hypotheses-Guided Self Distillation for Continual Personalizationcited by 1
  17. 2026Language Models Can Control Their Own Attentioncited by 1
  18. 2026Reason Through the Latent! Making Latent Visual Reasoning Necessarycited by 1
  19. 2026The Router Within: Eliciting Native Skill Routing from a Frozen LLMcited by 1
  20. 2026AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Modelscited by 1
  21. 2026Omni2Web: Benchmarking Audiovisual Website Developmentcited by 1
  22. 2026GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Modelscited by 1
  23. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  24. 2026CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agentscited by 0
  25. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  26. 2026TRACE-TS: Attribution-Grounded and Traceable Sensor-Language Reasoning for Human Activity Understandingcited by 0
  27. 2026Fighting Fire with Fire: On the Feasibility of Protecting Exercises Against AI Cheatingcited by 0
  28. 2026When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verificationcited by 0
  29. 2026Mind the Gaps: Mixture-of-Minds for Human Simulationcited by 0
  30. 2026Not an A11y: How Android Accessibility Exposes Mobile AI Agents to Indirect Prompt Injectioncited by 0
  31. 2026MADBench: A Benchmark for Modality-Aware Audio Deepfake Detectioncited by 0
  32. 2026Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuningcited by 0
  33. 2026Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Tracescited by 0
  34. 2026LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Taskscited by 0
  35. 2026RamseyGadgets: A Graph Construction Dataset for LLMscited by 0
  36. 2026BaT: Towards Self-Evolving Medical Research Agent with Stage Rubricscited by 0
  37. 2026Probing the Prefill: Detecting Code Vulnerabilities via Latent Activationscited by 0
  38. 2026Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signalscited by 0

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Gemma 3 Technical Report
  2. 2026Phi-4-reasoning-vision-15B Technical Report
  3. 2023On the Performance of Multimodal Language Models
  4. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  5. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
  6. 2026Toward Native Multimodal Modeling: A Roadmap
  7. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
  8. 2024PALO: A Polyglot Large Multimodal Model for 5B People