mlsanityData to 7 Oct 2026

LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling

arXiv 2605.00616 ↗First posted 1 May 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Realistic evaluation of LLM serving systems requires online workloads, dynamic arrivals, queueing, and the serving engine's local scheduling for execution batching, but running such experiments on GPUs is expensive. Existing simulators reduce this cost, but often operate offline or in time-warped mode, re-implement serving-engine schedulers, or require accurate operator/kernel-level latency models. We present LLM-Emu, a serving-native emulator for vLLM that preserves the production HTTP, scheduling, KV-cache, and output-processing paths while replacing only GPU forward execution with profile-sampled latency and synthetic output tokens. Tested on two different GPUs, four model variants, two model families, two attention backends, and both Poisson and bursty ShareGPT workloads, LLM-Emu closely tracks real vLLM serving behavior: TPOT and ITL stay within 4.8% absolute error, E2E latency within 5.3%, and output throughput within 1.9%; TTFT is less stable, with maximum error 10.4%, reflecting its sensitivity to admission and queue state. These results suggest that lightweight, serving-native emulation can support practical online experimentation for LLM-serving systems. LLM-Emu is open sourced at https://github.com/AKafakA/llm-emu.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 13 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

12 of 13 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving9 Jan
  2. 2025Frontier: Simulating the Next Generation of LLM Inference Systems5 Aug
  3. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving26 Nov
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 6

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  5. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
  6. 2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  2. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  3. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  6. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  7. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  8. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving