mlsanityData to 14 Oct 2026

DeepSeek-V3 Technical Report

arXiv 2412.19437 ↗First posted 27 Dec 2024cs.CL cs.AIcited by 811

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

Key facts. Cited by 811 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 49 of its 104 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

49 of 104 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks19 Dec
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism5 Jan
  3. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 46

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
34 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  15. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  16. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  17. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  18. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  19. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  20. 2024Massive Activations in Large Language Modelscited by 294
  21. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  22. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  23. 2022FP8 Formats for Deep Learningcited by 196
  24. 2024Are We Done with MMLU?cited by 194
  25. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  26. 2023Microscaling Data Formats for Deep Learningcited by 152
  27. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  28. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  29. 2024Scaling FP8 training to trillion-token LLMscited by 65
  30. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 43
  31. 2023Zero Bubble Pipeline Parallelismcited by 40
  32. 2024Fewer Truncations Improve Language Modelingcited by 35
  33. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  34. 2024Ascend HiFloat8 Format for Deep Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 435; 2026: 376.

811 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  3. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  5. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  6. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  7. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  9. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  10. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  11. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 110
  12. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
38 more
  1. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  2. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  3. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  6. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  7. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 82
  8. 2025Efficient Reasoning with Hidden Thinkingcited by 75
  9. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 74
  10. 2025Distillation Scaling Lawscited by 67
  11. 2025Reinforcement Learning from Human Feedbackcited by 58
  12. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  13. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 58
  14. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  15. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 56
  16. 2025Training a Scientific Reasoning Model for Chemistrycited by 56
  17. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 56
  18. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 54
  19. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 54
  20. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 53
  21. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 52
  22. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 50
  23. 2025VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thoughtcited by 49
  24. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  25. 2025START: Self-taught Reasoner with Toolscited by 47
  26. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 46
  27. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  28. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 44
  29. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  30. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  31. 2025Rethinking Reflection in Pre-Trainingcited by 43
  32. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  33. 2025Reasoning Language Models: A Blueprintcited by 42
  34. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42
  35. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  36. 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQLcited by 38
  37. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  38. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37

50 most cited of 811 All 811 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
  2. 2025A Review of DeepSeek Models' Key Innovative Techniques
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  4. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  5. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
  6. 2025On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
  7. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding