mlsanityData to 7 Oct 2026

Gemma 2: Improving Open Language Models at a Practical Size

arXiv 2408.00118 ↗First posted 31 Jul 2024cs.CL cs.AIcited by 1,928

Not on the map. Why?

Cited by 303 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Key facts. Cited by 1,928 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

34 of 47 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
22 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  10. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 600
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  13. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  14. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  15. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  16. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  17. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  18. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  19. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  20. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  21. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  22. 2023Towards Agile Text Classifiers for Everyonecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 288; 2025: 982; 2026: 658.

1,928 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  7. 2025Kimi K2: Open Agentic Intelligencecited by 523
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  11. 2025Muon is Scalable for LLM Trainingcited by 381
  12. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
38 more
  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  7. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  8. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  9. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  11. 2026Gemma 4 Technical Reportcited by 187
  12. 2026MiMo-V2-Flash Technical Reportcited by 163
  13. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  14. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  15. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  16. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  17. 2024BatchTopK Sparse Autoencoderscited by 126
  18. 2025BIG-Bench Extra Hardcited by 123
  19. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  20. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  21. 2025Training Agents Inside of Scalable World Modelscited by 122
  22. 2025Confidence Improves Self-Consistency in LLMscited by 111
  23. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  24. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  25. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 97
  26. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  27. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  28. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  29. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  30. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  31. 2024Scaling Laws for Precisioncited by 92
  32. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  33. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
  34. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  35. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  36. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  37. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  38. 2024How to Evaluate Reward Models for RLHFcited by 84

50 most cited of 1,928 All 1,928 (JSON)