mlsanityData to 14 Oct 2026

HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing

arXiv 2604.18529 ↗First posted 20 Apr 2026cs.PF cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 47 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

26 of 47 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  2. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits31 Oct
  3. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs2 Mar
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  8. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  9. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  10. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  11. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  12. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
3 more
  1. 2021LazyFormer: Self Attention with Lazy Updatecited by 16
  2. 2022Demystifying the Nvidia Ampere Architecture through Microbenchmarking and Instruction-level Analysiscited by 9
  3. 2024Exploring and Evaluating Real-world CXL: Use Cases and System Adoptioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 2
  2. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  2. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  3. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  4. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  5. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  6. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  7. 2025Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU
  8. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference