mlsanityData to 7 Oct 2026

SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling

arXiv 2503.02550 ↗First posted 4 Mar 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Deep Learning (DL), especially with Large Language Models (LLMs), brings benefits to various areas. However, DL training systems usually yield prominent idling GPU resources due to many factors, such as resource allocation and collective communication. To improve GPU utilization, we present SpecInF, which adopts a Speculative Inference Filling method to exploit idle GPU resources. It collocates each primary training instance with additional inference instances on the same GPU, detects the training bubbles and adaptively fills with online or offline inference workloads. Our results show that SpecInF can effectively enhance GPU utilization under mainstream parallel training modes, delivering additional up to 14× offline inference throughputs than TGS and 67% reduction in online inference p95 latency than MPS, while guaranteeing collocated training throughput.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 16 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

5 of 16 references.

References · 5

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  3. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  4. 2023FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inferencecited by 6
  5. 2021Efficient Strong Scaling Through Burst Parallel Trainingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  2. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
  3. 2025SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  5. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
  6. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  7. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  8. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices