mlsanityData to 14 Oct 2026

DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models

arXiv 2507.09955 ↗First posted 14 Jul 2025cs.AIcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

DeepSeek, a Chinese Artificial Intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and open-source advantages. This paper begins by reviewing the evolution of large AI models focusing on paradigm shifts, the mainstream Large Language Model (LLM) paradigm, and the DeepSeek paradigm. Subsequently, the paper highlights novel algorithms introduced by DeepSeek, including Multi-head Latent Attention (MLA), Mixture-of-Experts (MoE), Multi-Token Prediction (MTP), and Group Relative Policy Optimization (GRPO). The paper then explores DeepSeek engineering breakthroughs in LLM scaling, training, inference, and system-level optimization architecture. Moreover, the impact of DeepSeek models on the competitive AI landscape is analyzed, comparing them to mainstream LLMs across various fields. Finally, the paper reflects on the insights gained from DeepSeek innovations and discusses future trends in the technical and engineering development of large AI models, particularly in data, training, and reasoning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 97 of its 144 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

97 of 144 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 92

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2017Proximal Policy Optimization Algorithmscited by 15,215
  8. 2024The Llama 3 Herd of Modelscited by 14,715
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
38 more
  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  13. 2023Mistral 7Bcited by 3,164
  14. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  16. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  17. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  18. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  19. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  20. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  21. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  22. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  23. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  24. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  25. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  26. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  27. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  28. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  29. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  30. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  31. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  32. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  33. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  34. 2023BloombergGPT: A Large Language Model for Financecited by 939
  35. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  36. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  37. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  38. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Modelscited by 3
  2. 2025Robust and Efficient Communication in Multi-Agent Reinforcement Learningcited by 2
  3. 2026Multi-Agent Systems: From Classical Paradigms to Large Foundation Model-Enabled Futurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Review of DeepSeek Models' Key Innovative Techniques
  2. 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  4. 2025Large Language Model enabled Mathematical Modeling
  5. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
  6. 2025Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis
  7. 2025Brief analysis of DeepSeek R1 and its implications for Generative AI
  8. 2025A Showdown of ChatGPT vs DeepSeek in Solving Programming Tasks