# Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

arXiv: https://arxiv.org/abs/2507.19427 · HTML: https://mlsanity.com/paper/2507.19427/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://mlsanity.com/paper/2406.06858/) (2024)
- [Upcycling Large Language Models into Mixture of Experts](https://mlsanity.com/paper/2410.07524/) (2024)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Serving Large Language Models on Huawei CloudMatrix384](https://mlsanity.com/paper/2506.12708/) (2025)
- [InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers](https://mlsanity.com/paper/2502.03885/) (2025)

## Built on by (most cited first, 30 of 55)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [DeepSeek-OCR 2: Visual Causal Flow](https://mlsanity.com/paper/2601.20552/) (2026)
- [NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale](https://mlsanity.com/paper/2508.10711/) (2025)
- [BabyVision: Visual Reasoning Beyond Language](https://mlsanity.com/paper/2601.06521/) (2026)
- [Step-Audio-R1 Technical Report](https://mlsanity.com/paper/2511.15848/) (2025)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline](https://mlsanity.com/paper/2603.01050/) (2026)
- [RollArt: Disaggregated Multi-Task Agentic RL Training at Scale](https://mlsanity.com/paper/2512.22560/) (2025)
- [Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving](https://mlsanity.com/paper/2509.17863/) (2025)
- [Hyperparameter Transfer with Mixture-of-Expert Layers](https://mlsanity.com/paper/2601.20205/) (2026)
- [SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?](https://mlsanity.com/paper/2602.03916/) (2026)
- [GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts](https://mlsanity.com/paper/2509.25160/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving](https://mlsanity.com/paper/2602.06502/) (2026)
- [Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers](https://mlsanity.com/paper/2512.03870/) (2025)
- [PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching](https://mlsanity.com/paper/2602.00509/) (2026)
- [Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems](https://mlsanity.com/paper/2602.09721/) (2026)
- [xLLM Technical Report](https://mlsanity.com/paper/2510.14686/) (2025)
- [Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework](https://mlsanity.com/paper/2509.26534/) (2025)
- [Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction](https://mlsanity.com/paper/2511.23476/) (2025)
- [Role-Based Fault Tolerance System for LLM RL Post-Training](https://mlsanity.com/paper/2512.22492/) (2025)
- [Making MoE-based LLM Inference Resilient with Tarragon](https://mlsanity.com/paper/2601.01310/) (2026)
- [Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads](https://mlsanity.com/paper/2601.21351/) (2026)
- [SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs](https://mlsanity.com/paper/2604.23996/) (2026)
- [Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design](https://mlsanity.com/paper/2606.10493/) (2026)
- [FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication](https://mlsanity.com/paper/2508.03760/) (2025)
- [Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates](https://mlsanity.com/paper/2508.16151/) (2025)
- [MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training](https://mlsanity.com/paper/2511.21431/) (2025)

## Abstract (arXiv)

Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces both KV cache size and computation while maintaining high attention expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed inference system that decouples attention and Feed-Forward Network (FFN) layers into specialized subsystems. This co-design achieves unprecedented cost efficiency: Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at longer context. Step-3 achieves low cost while activating 38B parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness. We perform a head-to-head comparison with DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs achieves a decoding throughput of up to 4,039 tokens per second per GPU under 50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324 in the same setup and sets a new Pareto frontier for LLM decoding.