mlsanityData to 14 Oct 2026

Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

arXiv 2507.19427 ↗First posted 25 Jul 2025cs.LG cs.AIcited by 55

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces both KV cache size and computation while maintaining high attention expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed inference system that decouples attention and Feed-Forward Network (FFN) layers into specialized subsystems. This co-design achieves unprecedented cost efficiency: Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at longer context. Step-3 achieves low cost while activating 38B parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness. We perform a head-to-head comparison with DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs achieves a decoding throughput of up to 4,039 tokens per second per GPU under 50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324 in the same setup and sets a new Pareto frontier for LLM decoding.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 11 of its 24 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

11 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
  2. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures14 May
  3. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct

Other references · 8

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  6. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  7. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  8. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 35.

55 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026DeepSeek-OCR 2: Visual Causal Flowcited by 74
  3. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  4. 2026BabyVision: Visual Reasoning Beyond Languagecited by 49
  5. 2025Step-Audio-R1 Technical Reportcited by 36
  6. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 35
  7. 2026STEP3-VL-10B Technical Reportcited by 32
  8. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  9. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
  10. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  11. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Servingcited by 13
  12. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
38 more
  1. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?cited by 11
  2. 2025GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contextscited by 8
  3. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  4. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  5. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  6. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  7. 2026Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systemscited by 4
  8. 2025xLLM Technical Reportcited by 3
  9. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  10. 2025Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interactioncited by 2
  11. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  12. 2026Making MoE-based LLM Inference Resilient with Tarragoncited by 2
  13. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloadscited by 2
  14. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 2
  15. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 2
  16. 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 1
  17. 2025Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substratescited by 1
  18. 2025MemFine: Memory-Aware Fine-Grained Scheduling for MoE Trainingcited by 1
  19. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelismcited by 1
  20. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  21. 2026S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balancecited by 1
  22. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmarkcited by 1
  23. 2026CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregationcited by 1
  24. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernelscited by 1
  25. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0
  26. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Groundingcited by 0
  27. 2026TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settingscited by 0
  28. 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approachcited by 0
  29. 2026True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Liescited by 0
  30. 2026Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregationcited by 0
  31. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Servingcited by 0
  32. 2026DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelismcited by 0
  33. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0
  34. 2026VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfercited by 0
  35. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0
  36. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  37. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  38. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inferencecited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
  2. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
  3. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  4. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
  5. 2025Hardware-Efficient Attention for Fast Decoding
  6. 2024A Thorough Examination of Decoding Methods in the Era of LLMs
  7. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  8. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding