mlsanityData to 14 Oct 2026

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

arXiv 2608.06723 ↗First posted 7 Aug 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 31 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

19 of 31 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2023BloombergGPT: A Large Language Model for Financecited by 939
7 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  3. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  4. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7
  5. 2025Preserving Privacy and Utility in LLM-Based Product Recommendationscited by 5
  6. 2026HDLFORGE: A Two-Stage Multi-Agent Framework for Efficient Verilog Code Generation with Adaptive Model Escalationcited by 4
  7. 2026This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  2. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  5. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  6. 2023Efficient LLM inference solution on Intel GPU
  7. 2025System-performance and cost modeling of Large Language Model training and inference
  8. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models