Data to 14 Oct 2026

Vidur: A Large-Scale Simulation Framework For LLM Inference

arXiv 2405.05465 ↗First posted 8 May 2024cs.LG cs.AI cs.CLcited by 117

Not on the map. Why?

Cited by 58 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Optimizing the deployment of Large language models (LLMs) is expensive today since it requires experimentally running an application workload against an LLM implementation while exploring large configuration space formed by system knobs such as parallelization strategies, batching techniques, and scheduling policies. To address this challenge, we present Vidur - a large-scale, high-fidelity, easily-extensible simulation framework for LLM inference performance. Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates the end-to-end inference performance for different workloads by estimating several metrics of interest such as latency and throughput. We validate the fidelity of Vidur on several LLMs and show that it estimates inference latency with less than 9% error across the range. Further, we present Vidur-Search, a configuration search tool that helps optimize LLM deployment. Vidur-Search uses Vidur to automatically identify the most cost-effective deployment configuration that meets application performance constraints. For example, Vidur-Search finds the best deployment configuration for LLaMA2-70B in one hour on a CPU machine, in contrast to a deployment-based exploration which would require 42K GPU hours - costing ~218K dollars. Source code for Vidur is available at https://github.com/microsoft/vidur.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

15 of 30 references.

References · 15

2017Attention Is All You Needcited by 60,361
2020Language Models are Few-Shot Learnerscited by 32,419
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
2014cuDNN: Efficient Primitives for Deep Learningcited by 245
2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
2020Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Trainingcited by 22

3 more

2022Building a Performance Model for Deep Learning Recommendation Model Training on GPUscited by 10
2023Proteus: Simulating the Performance of Distributed DNN Trainingcited by 10
2023Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpuscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 35; 2026: 68.

117 papers.

Sort:

By area:

2024On Large Language Model Continual Unlearningcited by 52
2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 19
2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 19
2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 19
2024Very Large-Scale Multi-Agent Simulation in AgentScopecited by 18
2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 18
2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16

38 more

2025Glia: A Human-Inspired AI for Automated Systems Design and Optimizationcited by 16
2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systemscited by 10
2025On Evaluating Performance of LLM Inference Serving Systemscited by 10
2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 10
2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
2025Niyama : Breaking the Silos of LLM Inference Servingcited by 7
2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7
2025LLM Serving Optimization with Variable Prefill and Decode Lengthscited by 7
2026Improving Coherence and Persistence in Agentic AI for System Optimizationcited by 7
2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 6
2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
2025Compass: SLO-aware Query Planner for Compound AI Serving at Scalecited by 5
2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Servingcited by 5
2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5
2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanismcited by 3
2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inferencecited by 2
2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2

50 most cited of 117 All 117 (JSON)