mlsanityData to 14 Oct 2026

Llemma: An Open Language Model For Mathematics

arXiv 2310.10631 ↗First posted 16 Oct 2023cs.CL cs.AI cs.LOcited by 388

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.

Key facts. Cited by 388 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 63 of its 92 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

63 of 92 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text10 Oct
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  3. 2022A Survey of Deep Learning for Mathematical Reasoning20 Dec
  4. 2022Teaching Algorithmic Reasoning via In-context Learning15 Nov
  5. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun
  6. 2020Generative Language Modeling for Automated Theorem Proving7 Sep

Other references · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  14. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  15. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  17. 2023BloombergGPT: A Large Language Model for Financecited by 939
  18. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  19. 2022Galactica: A Large Language Model for Sciencecited by 744
  20. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  21. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  22. 2022PAL: Program-aided Language Modelscited by 652
  23. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  24. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  25. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  26. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  27. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  28. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  29. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  30. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 291
  31. 2022Autoformalization with Large Language Modelscited by 263
  32. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  33. 2022HyperTree Proof Search for Neural Theorem Provingcited by 197
  34. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  35. 2023SantaCoder: don't reach for the stars!cited by 170
  36. 2022Formal Mathematics Statement Curriculum Learningcited by 142
  37. 2019The Lean mathematical librarycited by 134
  38. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 132

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 194; 2025: 118; 2026: 67.

388 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2024Mixtral of Expertscited by 1,823
  4. 2025s1: Simple test-time scalingcited by 1,341
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  8. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  10. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  12. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
38 more
  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
  3. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  4. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 197
  5. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  6. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  7. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  9. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  10. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  11. 2024Continual Learning for Large Language Models: A Surveycited by 147
  12. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  13. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  14. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  15. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  16. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
  17. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  18. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  19. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  20. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  21. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  22. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 102
  23. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  24. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  25. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  26. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  27. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  28. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
  29. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  30. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  31. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  32. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 79
  33. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  34. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  35. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  36. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  37. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  38. 2024MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Modelscited by 70

50 most cited of 388 All 388 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMs
  2. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
  3. 2025Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
  4. 2026MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics
  5. 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofs
  6. 2026MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis
  7. 2025TopoAlign: A Framework for Aligning Code to Math via Topological Decomposition
  8. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs