mlsanityData to 7 Oct 2026

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

arXiv 2606.19348 ↗First posted 26 Apr 2026cs.CL cs.AIcited by 877

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-Constrained Hyper-Connections (mHC) that enhance conventional residual connections; (3) and the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible. The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4.

Key facts. Cited by 877 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. First posted on arXiv 26 Apr 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 877.

877 papers.

Sort:

By area:

  1. 2026Gemma 4 Technical Reportcited by 187
  2. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  3. 2026Kimi K3: Open Frontier Intelligencecited by 99
  4. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  5. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  6. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
  7. 2026MiniMax Sparse Attentioncited by 34
  8. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  9. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  10. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 29
  11. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  12. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 25
38 more
  1. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  2. 2026Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agentscited by 22
  3. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
  4. 2026SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revisioncited by 21
  5. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 20
  6. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 19
  7. 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
  8. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 17
  9. 2026On the Geometry of On-Policy Distillationcited by 17
  10. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 16
  11. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  12. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  13. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  14. 2026Trajectory-Refined Distillationcited by 14
  15. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 14
  16. 2026Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectoriescited by 13
  17. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Draftingcited by 12
  18. 2026Rubric-based On-policy Distillationcited by 11
  19. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  20. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  21. 2026Policy and World Modeling Co-Training for Language Agentscited by 11
  22. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  23. 2026GenClaw: Code-Driven Agentic Image Generationcited by 10
  24. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  25. 2026GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?cited by 10
  26. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  27. 2026On-policy Distillation with Verifiable Rewardcited by 10
  28. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 9
  29. 2026OPRD: On-Policy Representation Distillationcited by 9
  30. 2026Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Samplingcited by 9
  31. 2026DOPD: Dual On-policy Distillationcited by 9
  32. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memorycited by 9
  33. 2026Code World Model: Coding Agent as World Braincited by 9
  34. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  35. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  36. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 8
  37. 2026Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluationcited by 8
  38. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8

50 most cited of 877 All 877 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  3. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
  4. 2024DeepSeek-V3 Technical Report
  5. 2025MiniMax-01: Scaling Foundation Models with Lightning Attention
  6. 2025Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis
  7. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding