mlsanityData to 7 Oct 2026

TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling

arXiv 2509.03377 ↗First posted 3 Sep 2025cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but its link and device-side DDR bandwidth are far below HBM, so decoding stalls once traffic shifts to the CXL tier. Many CXL controllers are starting to add generic lossless compression, yet applying commodity codecs directly to standard word-major LLM tensors is largely ineffective, especially for token-major KV streams. We propose TRACE (Traffic-Reduced Architecture for Compression and Elasticity), which preserves the unmodified CXL.mem interface but changes the device-internal representation. It stores tensors in a channel-major, disaggregated bit-plane layout, and applies a KV-specific transform before compression, converting mixed-field words into low-entropy plane streams that commodity codecs can compress. The same substrate enables precision-proportional fetch by reading only the required bit-planes. Across public LLMs, TRACE reduces BF16 weight footprint by 25.2% and BF16 KV footprint by 46.9% losslessly, with per-layer KV ratios peaking at 2.69×. In trace-driven system modeling, once KV spills to CXL, GPT-OSS-120B-MXFP4 improves throughput at 128k tokens from 16.28 to 68.99 tok/s (4.24×). DRAMSim3 shows up to 40.3% lower DRAM access energy under plane-aligned fetch. A 7,nm SystemVerilog implementation sustains 256,GB/s device bandwidth. Relative to a CXL controller with generic inline lossless compression, TRACE only adds 7.2% area, 4.7% power, and 6.0% load-to-use latency at 2,GHz and 0.7,V.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 42 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

23 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2024Mixtral of Expertscited by 1,785
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
10 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  4. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  5. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  7. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  8. 2023An Introduction to the Compute Express Link (CXL) Interconnectcited by 9
  9. 2024SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantizationcited by 2
  10. 2024ZipCache: A DRAM/SSD Cache with Built-in Transparent Compressioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
  2. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  3. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  4. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
  5. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  6. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  7. 2026The KV Cache Is the New Memory Wall
  8. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading