mlsanityData to 14 Oct 2026

PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers

arXiv 2411.19114 ↗First posted 28 Nov 2024cs.DC cs.AI cs.ARcited by 0

Not on the map. Why?

Abstract (arXiv)

NVIDIA's Multi-Instance GPU (MIG) is a feature that enables system designers to reconfigure one large GPU into multiple smaller GPU slices. This work characterizes this emerging GPU and evaluates its effectiveness in designing high-performance AI inference servers. Our study reveals that the data preprocessing stage of AI inference causes significant performance bottlenecks to MIG. To this end, we present PREBA, which is a hardware/software co-design targeting MIG inference servers. Our first proposition is an FPGA-based data preprocessing accelerator that unlocks the full potential of MIG with domain-specific acceleration of data preprocessing. The MIG inference server unleashed from preprocessing overheads is then augmented with our dynamic batching system that enables high-performance inference. PREBA is implemented end-to-end in real systems, providing a 3.7x improvement in throughput, 3.4x reduction in tail latency, 3.5x improvement in energy-efficiency, and 3.0x improvement in cost-efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 71 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

18 of 71 references.

References · 18

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  4. 2019MLPerf Inference Benchmarkcited by 199
  5. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  6. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  7. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  8. 2019PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inferencecited by 61
  9. 2020Analyzing and Mitigating Data Stalls in DNN Trainingcited by 36
  10. 2021Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognitioncited by 34
  11. 2018Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networkscited by 32
  12. 2019PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Unitscited by 30
6 more
  1. 2021Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Trainingcited by 28
  2. 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 22
  3. 2022MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learningcited by 15
  4. 2022An Analysis of Collocation on GPUs for Deep Learning Trainingcited by 6
  5. 2021FlexTOE: Flexible TCP Offload with Fine-Grained Parallelismcited by 4
  6. 2022PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Serverscited by 2

Built on by

Not cited yet.