# Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

arXiv: https://arxiv.org/abs/2607.23115 · HTML: https://mlsanity.com/paper/2607.23115/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 81 references that are in the map.
- First posted on arXiv 25 Jul 2026.

## Builds on (most cited first, 30 of 32)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Kernel Operations on the GPU, with Autodiff, without Memory Overflows](https://mlsanity.com/paper/2004.11127/) (2020)
- [GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge](https://mlsanity.com/paper/2201.07705/) (2022)
- [CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters](https://mlsanity.com/paper/2308.00852/) (2023)
- [SkyServe: Serving AI Models across Regions and Clouds with Spot Instances](https://mlsanity.com/paper/2411.01438/) (2024)
- [Bringing UMAP Closer to the Speed of Light with GPU Acceleration](https://mlsanity.com/paper/2008.00325/) (2020)
- [Automatic Curriculum Generation for Learning Adaptation in Networking](https://mlsanity.com/paper/2202.05940/) (2022)
- [THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression](https://mlsanity.com/paper/2302.08545/) (2023)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [Whisper: Fast Flooding for Low-Power Wireless Networks](https://mlsanity.com/paper/1809.03699/) (2018)
- [AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains](https://mlsanity.com/paper/2407.21255/) (2024)
- [Multi-Agent Collaborative Inference via DNN Decoupling: Intermediate Feature Compression and Edge Learning](https://mlsanity.com/paper/2205.11854/) (2022)
- [SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads](https://mlsanity.com/paper/2312.16733/) (2023)
- [Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads](https://mlsanity.com/paper/2410.07381/) (2024)
- [The Case for Task Sampling based Learning for Cluster Job Scheduling](https://mlsanity.com/paper/2108.10464/) (2021)
- [Toward Performance-Portable PETSc for GPU-based Exascale Systems](https://mlsanity.com/paper/2011.00715/) (2020)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [Eva: Cost-Efficient Cloud-Based Cluster Scheduling](https://mlsanity.com/paper/2503.07437/) (2025)
- [Kernelet: High-Throughput GPU Kernel Executions with Dynamic Slicing and Scheduling](https://mlsanity.com/paper/1303.5164/) (2013)
- [OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation](https://mlsanity.com/paper/2310.02422/) (2023)
- [A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro](https://mlsanity.com/paper/2409.19488/) (2024)
- [Porting WarpX to GPU-accelerated platforms](https://mlsanity.com/paper/2101.12149/) (2021)

## Abstract (arXiv)

This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency and up to 1.79 times higher system throughput.