mlsanityData to 7 Oct 2026

Efficient, VRAM-Constrained xLM Inference on Clients

arXiv 2604.26334 ↗First posted 29 Apr 2026cs.DC cs.AR cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

To usher in the next round of client AI innovation, there is an urgent need to enable efficient, lossless inference of high-accuracy large language models (LLMs) and vision language models (VLMs), jointly referred to as xLMs, on client systems. To address this, we present pipelined sharding, a novel, benchmark-profile-guided CPU-GPU hybrid scheduling technique to achieve efficient, VRAM-constrained inference for both dense and mixture-of-experts (MoE) LLMs. Using a combination of model sharding at the sub-layer level, CPU offloading, pipelined copy-compute, and prioritized tensor placement in VRAM, it optimizes both time-to-first-token (TTFT) and tokens per second (TPS) metrics, while flexibly adapting to system and inference conditions. For efficient, high-accuracy VLM inference, we combine pipelined sharding with a llama.cpp implementation of three well-understood prior ideas (jointly called VLMOpt), namely, vision tensor CPU offloading, flash attention, and vision and language model VRAM overlap avoidance. These enhancements are targeted at improving client xLM inference in future releases of two important NVIDIA products - the In-Game Inferencing software development kit (IGI SDK) and the Cosmos-Reason1 (CR1) physical AI reasoning VLM. Highlights from our rigorous evaluation spanning multiple models and client systems include: for interactive use, TTFT improves by up to 6.7x and TPS by up to 30x for LLMs, and CR1 inference's VRAM demand is down by 10x, while in batched mode, throughput improves by up to 8.2x, all compared to their respective aggressive baselines. This paper is accepted at the 9th MLSys Conference (Industry Track), 2026. Code and artifact available at: https://github.com/deepshnv/pipeshard-mlsys26-ae

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 34 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

18 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024FastVLM: Efficient Vision Encoding for Vision Language Models17 Dec

Other references · 17

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  12. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
5 more
  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  2. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  3. 2023LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Pluginscited by 49
  4. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  5. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
  2. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
  5. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  6. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
  7. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  8. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs