mlsanityData to 7 Oct 2026

MLPerf Inference Benchmark

arXiv 1911.02549 ↗First posted 6 Nov 2019cs.LG cs.PF stat.MLcited by 195

Not on the map. Why?

Abstract (arXiv)

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 25; 2023: 31; 2024: 27; 2025: 28; 2026: 62.

195 papers.

Sort:

By area:

  1. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  2. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  3. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  4. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  7. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  8. 2021MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluationcited by 88
  9. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  10. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  11. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  12. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
38 more
  1. 2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
  2. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  3. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  4. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  5. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28
  6. 2023NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systemscited by 27
  7. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  8. 2021Anchor-based Plain Net for Mobile Image Super-Resolutioncited by 24
  9. 2023HPC-GPT: Integrating Large Language Model for High-Performance Computingcited by 22
  10. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  11. 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inferencecited by 21
  12. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scalingcited by 19
  13. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  14. 2025SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Modelcited by 18
  15. 2021AI Accelerator Survey and Trendscited by 17
  16. 2023Unveiling Energy Efficiency in Deep Learning: Measurement, Prediction, and Scoring across Edge Devicescited by 16
  17. 2021MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systemscited by 15
  18. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 15
  19. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  20. 2025Mixture of Experts in Large Language Modelscited by 15
  21. 2022XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaversecited by 14
  22. 2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
  23. 2025CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learningcited by 13
  24. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Modelscited by 12
  25. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11
  26. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  27. 2021On the Accuracy of Analog Neural Network Inference Acceleratorscited by 9
  28. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  29. 2023Design Principles for Lifelong Learning AI Acceleratorscited by 9
  30. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  31. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  32. 2022DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloadscited by 8
  33. 2023MoCA: Memory-Centric, Adaptive Execution for Multi-Tenant Deep Neural Networkscited by 8
  34. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  35. 2025Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practicescited by 8
  36. 2021Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scalecited by 7
  37. 2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
  38. 2024Event-based Optical Flow on Neuromorphic Processor: ANN vs. SNN Comparison based on Activation Sparsificationcited by 7

50 most cited of 195 All 195 (JSON)