mlsanityData to 14 Oct 2026

Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs

arXiv 2512.20210 ↗First posted 23 Dec 2025cs.DCcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The serverless computing paradigm offers compelling advantages for deploying Large Language Model (LLM) inference services, including elastic scaling and pay-per-use billing. However, serving multiple fine-tuned LLMs via Low-Rank Adaptation (LoRA) in serverless environments faces critical challenges: reactive adapter loading causes significant cold start latency, and frequent adapter swapping leads to severe GPU memory fragmentation. In this paper, we present Predictive-LoRA (P-LoRA), a proactive and fragmentation-aware serverless inference system for LoRA-based LLMs. P-LoRA introduces two key innovations: (1) a lightweight LSTM-based traffic predictor that forecasts adapter demand and proactively prefetches hot adapters from host memory to GPU, reducing cold start latency by up to 68%; and (2) a page-based adapter memory management mechanism inspired by operating system virtual memory, which keeps GPU memory utilization above 87% even under heterogeneous adapter ranks. We evaluate P-LoRA using production-like workloads derived from the Azure Functions trace. Experimental results demonstrate that P-LoRA achieves 1.52x higher throughput than S-LoRA while reducing the average Time-To-First-Token (TTFT) by 35% under high concurrency scenarios.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 15 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

12 of 15 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Mistral 7Bcited by 3,164
  8. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  9. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  10. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  11. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  12. 2024GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitchingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Structured Prompt Optimization for Few-Shot Text Classification via Semantic Alignment in Latent Spacecited by 0
  2. 2026PreFT: Prefill-only finetuning for efficient inferencecited by 0
  3. 2026Stable Token-level Scheduling for Serverless LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
  2. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
  3. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  4. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
  5. 2026InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models ↗
  6. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  7. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  8. 2026CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration