mlsanityData to 7 Oct 2026

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

arXiv 2411.00136 ↗First posted 31 Oct 2024cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 45 of its 75 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

45 of 75 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2023Mistral 7Bcited by 3,065
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2024Qwen2 Technical Reportcited by 2,076
33 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2024Mixtral of Expertscited by 1,785
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
  13. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  14. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  15. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  16. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  17. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  18. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  19. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  20. 2021Efficient Attentions for Long Document Summarizationcited by 342
  21. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  22. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  23. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  24. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  25. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  26. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  27. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  28. 2022FP8 Quantization: The Power of the Exponentcited by 87
  29. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  30. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  31. 2023VCSUM: A Versatile Chinese Meeting Summarization Datasetcited by 18
  32. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  33. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 17.

31 papers.

Sort:

By area:

  1. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 73
  2. 2025Unified Multimodal Discrete Diffusioncited by 46
  3. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
  4. 2026MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observabilitycited by 6
  5. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 6
  6. 2025Curse of High Dimensionality Issue in Transformer for Long-context Modelingcited by 5
  7. 2025SymRAG: Efficient Neuro-Symbolic Retrieval Through Adaptive Query Routingcited by 5
  8. 2025FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Accesscited by 5
  9. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughputcited by 4
  10. 2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
  11. 2026Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Costcited by 4
  12. 2025From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agentscited by 2
19 more
  1. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  3. 2025On the Sustainability of AI Inferences in the Edgecited by 1
  4. 2026RLHFless: Serverless Computing for Efficient RLHFcited by 1
  5. 2026Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzingcited by 1
  6. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  7. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  8. 2025Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Indexcited by 0
  9. 2025Semantic-Aware Fuzzing: An Empirical Framework for LLM-Guided, Reasoning-Driven Input Mutationcited by 0
  10. 2025Streaming Tensor Programs: A Streaming Abstraction for Dynamic Parallelismcited by 0
  11. 2025Viability and Performance of a Private LLM Server for SMBs: A Benchmark Analysis of Qwen3-30B on Consumer-Grade Hardwarecited by 0
  12. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  13. 2026FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understandingcited by 0
  14. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Studycited by 0
  15. 2026Scalable AI Inference: Performance Analysis and Optimization of AI Model Servingcited by 0
  16. 2026Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Acceleratorscited by 0
  17. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  18. 2026FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Modelscited by 0
  19. 2026Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testingcited by 0