# MLPerf Inference Benchmark

arXiv: https://arxiv.org/abs/1911.02549 · HTML: https://mlsanity.com/paper/1911.02549/

## Key facts

- Cited by 199 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 30 of 199)

- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference](https://mlsanity.com/paper/2308.12066/) (2023)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation](https://mlsanity.com/paper/2110.01406/) (2021)
- [S3: Increasing GPU Utilization during Generative Inference for Higher Throughput](https://mlsanity.com/paper/2306.06000/) (2023)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning](https://mlsanity.com/paper/2107.08147/) (2021)
- [SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention](https://mlsanity.com/paper/2406.15486/) (2024)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)
- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendation](https://mlsanity.com/paper/2201.10095/) (2022)
- [NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems](https://mlsanity.com/paper/2304.04640/) (2023)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [Anchor-based Plain Net for Mobile Image Super-Resolution](https://mlsanity.com/paper/2105.09750/) (2021)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [HPC-GPT: Integrating Large Language Model for High-Performance Computing](https://mlsanity.com/paper/2311.12833/) (2023)
- [TokenPowerBench: Benchmarking the Power Consumption of LLM Inference](https://mlsanity.com/paper/2512.03024/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling](https://mlsanity.com/paper/2501.08219/) (2025)
- [SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model](https://mlsanity.com/paper/2505.22126/) (2025)
- [AI Accelerator Survey and Trends](https://mlsanity.com/paper/2109.08957/) (2021)
- [Unveiling Energy Efficiency in Deep Learning: Measurement, Prediction, and Scoring across Edge Devices](https://mlsanity.com/paper/2310.18329/) (2023)
- [T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives](https://mlsanity.com/paper/2401.16677/) (2024)
- [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024)

## Abstract (arXiv)

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.