mlsanityData to 14 Oct 2026

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

arXiv 2401.02954 ↗First posted 5 Jan 2024cs.CL cs.AI cs.LGcited by 642

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

Key facts. Cited by 642 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 60 of its 71 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

60 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  14. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  15. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  16. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  18. 2020GLU Variants Improve Transformercited by 1,804
  19. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  20. 2019Root Mean Square Layer Normalizationcited by 1,480
  21. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  22. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  23. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  24. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  25. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  26. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  27. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  28. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  29. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  30. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  31. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  32. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  33. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  34. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  35. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  36. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  37. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  38. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 201; 2025: 296; 2026: 145.

642 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2024DeepSeek-V3 Technical Reportcited by 811
  8. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024Yi: Open Foundation Models by 01.AIcited by 719
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
38 more
  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  2. 2025Muon is Scalable for LLM Trainingcited by 428
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  4. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  6. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  7. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 356
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  10. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  12. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  13. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  14. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  15. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  16. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  17. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  18. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  19. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  20. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  21. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  22. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  23. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  24. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  25. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  26. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  27. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  28. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  29. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  30. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  31. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  32. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  33. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  34. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  35. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  36. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 86
  37. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  38. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 81

50 most cited of 642 All 642 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models
  2. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  3. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  4. 2025A Review of DeepSeek Models' Key Innovative Techniques
  5. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond
  6. 2025Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipe
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs
  8. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models