mlsanityData to 14 Oct 2026

Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

arXiv 2603.10031 ↗First posted 27 Feb 2026cs.AR cs.AI cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present a cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs, benchmarking four models spanning 235B to 1 trillion parameters across three architectural families (MoE+MLA, Dense+GQA, MoE+GQA) on an 8-GPU cluster with 2TB aggregate HBM3e using vLLM v0.14.1. Our results demonstrate that architecture-aware optimization is essential: MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both. The AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations. A controlled AITER ablation on Llama-3.1-405B (n=5 per condition) reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically. Under text-only workloads, Llama-405B and DeepSeek V3.2 achieve comparable peak throughput (15,944 and 15,343 tok/s) despite an order-of-magnitude difference in active parameters. Under vision workloads, Qwen3-VL-235B reaches 47,873 tok/s, 6.5x higher than Kimi-K2.5 (7,327 tok/s). Active parameter count per token is associated with inference throughput, though confounded by differences in quantization, AITER acceleration, and tensor parallelism. All four models exhibit a common throughput saturation point consistent with a memory-bandwidth bottleneck (~500 concurrent for short sequences, ~100-200 for longer sequences). All models maintain 100% HTTP-level success rates through 1,000 concurrent users, processing 18.9 million tokens across 17,406 requests without failures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 33 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

27 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models24 Aug

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2025Qwen3-VL Technical Reportcited by 2,793
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2024Mixtral of Expertscited by 1,823
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
14 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2019MLPerf Inference Benchmarkcited by 199
  9. 2022FP8 Formats for Deep Learningcited by 196
  10. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  11. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  12. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  13. 2023Optimizing Distributed Training on Frontier for Large Language Modelscited by 12
  14. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
  2. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  3. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  4. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  5. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  6. 2026Dissecting GPU Utilization for LLM Inference on Nvidia Hopper
  7. 2026Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
  8. 2023Efficient LLM inference solution on Intel GPU