mlsanityData to 14 Oct 2026

GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines

arXiv 2502.15712 ↗First posted 22 Jan 2025cs.NI cs.AI cs.OScited by 1

Not on the map. Why?

Abstract (arXiv)

The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distributed serving rises and introduces much-dreaded network delays. In this paper, we investigate how the network can instead be a boon to the excessively high resource overheads of AI pipelines. To alleviate these overheads, we discuss how resource-intensive data processing tasks -- a key facet of growing AI pipeline complexity -- are well-matched for the computational characteristics of packet processing pipelines and how they can be offloaded onto SmartNICs. We explore the challenges and opportunities of offloading, and propose a research agenda for integrating network hardware into AI pipelines, unlocking new opportunities for optimization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 62 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

25 of 62 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2017The Kinetics Human Action Video Datasetcited by 1,909
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
13 more
  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  3. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  4. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  5. 2020Ekya: Continuous Learning of Video Analytics Models on Edge Compute Serverscited by 58
  6. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  7. 2021tf.data: A Machine Learning Data Processing Frameworkcited by 41
  8. 2020Analyzing and Mitigating Data Stalls in DNN Trainingcited by 36
  9. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  10. 2022IoT Anomaly Detection Methods and Applications: A Surveycited by 14
  11. 2020Taurus: A Data Plane Architecture for Per-Packet MLcited by 12
  12. 2022Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelinescited by 10
  13. 2024Tabular Learning: Encoding for Entity and Context Embeddingscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0