mlsanityData to 7 Oct 2026

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

arXiv 2405.04434 ↗First posted 7 May 2024cs.CL cs.AIcited by 1,093

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

Key facts. Cited by 1,093 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 47 of its 58 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

47 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
32 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  14. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  16. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  17. 2024Yi: Open Foundation Models by 01.AIcited by 706
  18. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  19. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  20. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  21. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  22. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  23. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  24. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  25. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  26. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  27. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  28. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  29. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  30. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  31. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  32. 2023Zero Bubble Pipeline Parallelismcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 212; 2025: 496; 2026: 385.

1,093 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  4. 2024DeepSeek-V3 Technical Reportcited by 724
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  7. 2025Kimi K2: Open Agentic Intelligencecited by 523
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  11. 2025Muon is Scalable for LLM Trainingcited by 381
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
38 more
  1. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  4. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  5. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  6. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  7. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  8. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  9. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  10. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  11. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  13. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  14. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  15. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  16. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  17. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  18. 2025Qwen3Guard Technical Reportcited by 125
  19. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  20. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  21. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  22. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  23. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  24. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  25. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  26. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  27. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 109
  28. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  29. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  30. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  31. 2026Kimi K3: Open Frontier Intelligencecited by 99
  32. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  33. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  34. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  35. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  36. 2025Flow Matching Policy Gradientscited by 86
  37. 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 82
  38. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82

50 most cited of 1,093 All 1,093 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2025Mixture of Experts in Large Language Models
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  5. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  6. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  7. 2026A.X K2 Technical Report
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Models