# Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider

arXiv: https://arxiv.org/abs/2003.03423 · HTML: https://mlsanity.com/paper/2003.03423/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 30 of 66)

- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems](https://mlsanity.com/paper/2401.17644/) (2024)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://mlsanity.com/paper/2505.09999/) (2025)
- [SkyServe: Serving AI Models across Regions and Clouds with Spot Instances](https://mlsanity.com/paper/2411.01438/) (2024)
- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs](https://mlsanity.com/paper/2312.05215/) (2023)
- [HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds](https://mlsanity.com/paper/2502.15524/) (2025)
- [Treehouse: A Case For Carbon-Aware Datacenter Software](https://mlsanity.com/paper/2201.02120/) (2022)
- [SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads](https://mlsanity.com/paper/2312.16733/) (2023)
- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale](https://mlsanity.com/paper/2608.00101/) (2026)
- [How Does It Function? Characterizing Long-term Trends in Production Serverless Workloads](https://mlsanity.com/paper/2312.10127/) (2023)
- [DAM: Towards A Foundation Model for Time Series Forecasting](https://mlsanity.com/paper/2407.17880/) (2024)
- [EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving](https://mlsanity.com/paper/2512.14946/) (2025)
- [FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters](https://mlsanity.com/paper/2510.11938/) (2025)
- [Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniques](https://mlsanity.com/paper/2211.15751/) (2022)
- [Trusted AI Agents in the Cloud](https://mlsanity.com/paper/2512.05951/) (2025)
- [A Deep Recurrent-Reinforcement Learning Method for Intelligent AutoScaling of Serverless Functions](https://mlsanity.com/paper/2308.05937/) (2023)
- [Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management](https://mlsanity.com/paper/2505.03756/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [CascadeServe: Unlocking Model Cascades for Inference Serving](https://mlsanity.com/paper/2406.14424/) (2024)
- [Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs](https://mlsanity.com/paper/2410.17840/) (2024)
- [Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs](https://mlsanity.com/paper/2512.20210/) (2025)
- [FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism](https://mlsanity.com/paper/2603.16514/) (2026)
- [Learning Cooperative Oversubscription for Cloud by Chance-Constrained Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.11759/) (2022)
- [OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud](https://mlsanity.com/paper/2311.12864/) (2023)
- [On a Foundation Model for Operating Systems](https://mlsanity.com/paper/2312.07813/) (2023)
- [Learned Best-Effort LLM Serving](https://mlsanity.com/paper/2401.07886/) (2024)

## Abstract (arXiv)

Function as a Service (FaaS) has been gaining popularity as a way to deploy computations to serverless backends in the cloud. This paradigm shifts the complexity of allocating and provisioning resources to the cloud provider, which has to provide the illusion of always-available resources (i.e., fast function invocations without cold starts) at the lowest possible resource cost. Doing so requires the provider to deeply understand the characteristics of the FaaS workload. Unfortunately, there has been little to no public information on these characteristics. Thus, in this paper, we first characterize the entire production FaaS workload of Azure Functions. We show for example that most functions are invoked very infrequently, but there is an 8-order-of-magnitude range of invocation frequencies. Using observations from our characterization, we then propose a practical resource management policy that significantly reduces the number of function coldstarts,while spending fewerresources than state-of-the-practice policies.