mlsanityData to 14 Oct 2026

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

arXiv 2505.09343 ↗First posted 14 May 2025cs.DC cs.AI cs.ARcited by 39

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model co-design can effectively address these challenges, enabling cost-efficient training and inference at scale. This paper presents an in-depth analysis of the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting key innovations such as Multi-head Latent Attention (MLA) for enhanced memory efficiency, Mixture of Experts (MoE) architectures for optimized computation-communication trade-offs, FP8 mixed-precision training to unlock the full potential of hardware capabilities, and a Multi-Plane Network Topology to minimize cluster-level network overhead. Building on the hardware bottlenecks encountered during DeepSeek-V3's development, we engage in a broader discussion with academic and industry peers on potential future hardware directions, including precise low-precision computation units, scale-up and scale-out convergence, and innovations in low-latency communication fabrics. These insights underscore the critical role of hardware and model co-design in meeting the escalating demands of AI workloads, offering a practical blueprint for innovation in next-generation AI systems.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 72 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

28 of 72 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
16 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  6. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  7. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  8. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  9. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  10. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  11. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  12. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  13. 2025TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitivescited by 24
  14. 2020An In-Depth Analysis of the Slingshot Interconnectcited by 13
  15. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  16. 2025UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecturecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 22.

39 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  2. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  3. 2025AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimizationcited by 23
  4. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 21
  5. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  6. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 12
  7. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  8. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  9. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  10. 2026Exploring Silent Data Corruption as a Reliability Challenge in LLM Trainingcited by 3
  11. 2026ToolCaching: Towards Efficient Caching for LLM Tool-callingcited by 2
  12. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUscited by 2
27 more
  1. 2026Practical FP4 Training for Large-Scale MoE Models on Hopper GPUscited by 2
  2. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 2
  3. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMscited by 1
  4. 2025VeriLocc: End-to-End Cross-Architecture Register Allocation via LLMcited by 1
  5. 2025A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attentioncited by 1
  6. 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 1
  7. 2025Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substratescited by 1
  8. 2025FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Errorcited by 1
  9. 2025Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracycited by 1
  10. 2025FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Modelscited by 1
  11. 2025From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports -- with Preliminary Extension to Lung Cancercited by 0
  12. 2025Approximate Multiplier Induced Error Propagation in Deep Neural Networkscited by 0
  13. 2026Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamilcited by 0
  14. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Acceleratorcited by 0
  15. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUscited by 0
  16. 2026Measuring Maximum Activations in Open Large Language Modelscited by 0
  17. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0
  18. 2026NASiC: 3D NAND-based CAM-Selected Multibit CIM Architecture for Efficient On-Device Mixture-of-Experts LLM Inferencecited by 0
  19. 2026MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systemscited by 0
  20. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  21. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  22. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpodscited by 0
  23. 2026FastTPS: An Optimized Method for LLM Token Phase for AI acceleratorscited by 0
  24. 2026DeaMoE: Efficient MoE Structure for Fast Small-Batch Decodingcited by 0
  25. 2026Mahalanobis-Based Multi-Head Attention for Complex State Propagationcited by 0
  26. 2026TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communicationcited by 0
  27. 2026KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  3. 2024DeepSeek-V3 Technical Report
  4. 2025Memory Analysis on the Training Course of DeepSeek Models
  5. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
  6. 2025A Review of DeepSeek Models' Key Innovative Techniques
  7. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs
  8. 2026SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD