mlsanityData to 14 Oct 2026

SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data

arXiv 2605.01060 ↗First posted 1 May 2026cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present SURGE, a streaming GPU encoding system deployed in production to generate embeddings for over 800 million texts across 40,000 logical partitions. Production embedding pipelines face a tension between logical data partitioning and efficient GPU utilization: processing each partition independently incurs P inter-process communication (IPC) calls whose overhead limits throughput for compute-light models. Our contributions are analytical: (i) a cost model (Theorem 1) predicting throughput within 2% across three encoders spanning a 15× parameter range; (ii) a memory-safety bound (Lemma 3) enabling a streaming two-threshold policy with peak memory O(Bmin + nmax) rather than O(N); and (iii) a φ/CV decision framework characterizing when the pattern applies beyond our workload. The naive fix of batching at fixed size requires O(N) peak memory (32.7 GB at 10M texts; infeasible beyond ~60M on 192 GB nodes), produces no output until all encoding completes, and offers no fault tolerance. SURGE achieves the same throughput with O(Bmin + nmax) bounded memory (2.6 GB), 68× faster time-to-first-output, and crash recovery at SuperBatch granularity. On 10M texts with 4 NVIDIA L4 GPUs, SURGE delivers 26,413 texts/s -- matching fixed-batch throughput while using 12.6× less memory. We validate on bge-base (109M, d=768, error 1.3%) and across log-normal σ in 1.0, 1.72, 2.5 (speedup invariant within ±3%), and compare against a partition-batched baseline (PB-PBP-LB), against which SURGE retains a 7% throughput edge and 2.5× faster TTFO. Complementary engineering -- zero-copy Arrow serialization (22-25× speedup) and async I/O pipelining (up to 93% benefit) -- realizes the design but is not the contribution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 45 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

15 of 45 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  6. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  9. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  11. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  12. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
3 more
  1. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  2. 2022Monolith: Real Time Recommendation System With Collisionless Embedding Tablecited by 30
  3. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28

Built on by

Not cited yet.