mlsanityData to 7 Oct 2026

Gemma: Open Models Based on Gemini Research and Technology

arXiv 2403.08295 ↗First posted 13 Mar 2024cs.CL cs.AIcited by 889

Not on the map. Why?

Cited by 181 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.

Key facts. Cited by 889 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
21 more
  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  15. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  17. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  18. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  19. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  20. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  21. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 343; 2025: 357; 2026: 189.

889 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  6. 2024What matters when building vision-language models?cited by 321
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  9. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  10. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
38 more
  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  2. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  3. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  7. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  8. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  9. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  10. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  11. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  12. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  13. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  14. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  15. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  16. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  17. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  18. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  19. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  20. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  21. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  22. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  23. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  24. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  25. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 93
  26. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  27. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  28. 2024Transformers need glasses! Information over-squashing in language taskscited by 87
  29. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  30. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  31. 2024On Verbalized Confidence Scores for LLMscited by 83
  32. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  33. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  34. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  35. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 78
  36. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  37. 2024Efficient multi-prompt evaluation of LLMscited by 72
  38. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72

50 most cited of 889 All 889 (JSON)