mlsanityData to 14 Oct 2026

Vortex: Bridging Extreme Compression and Efficient LLM Inference

arXiv 2609.12208 ↗First posted 10 Sep 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Extreme compression techniques, including vector quantization (VQ) and input-dependent sparsity, can significantly reduce the memory footprint of large language models (LLMs). However, a key challenge remains in translating such compression into practical efficiency. On conventional systolic-array-based accelerators, VQ incurs high dequantization overhead, while the irregular patterns of input-dependent sparsity are difficult to exploit. In this study, we address these challenges with Vortex, an architecture compatible with systolic-array-based accelerators with minimal hardware overhead, bridging the gap between extreme compression and efficient inference. Vortex adopts a bi-flow execution strategy that efficiently supports vector-quantized models across both prefill and decoding workloads, and we further optimize it through systematic design space exploration. On the algorithm side, we propose codebook-wise contextual sparsity to align with VQ execution. Across end-to-end workloads, Vortex achieves 8.03×--23.7× speedup and 5.68×--12.5× energy reduction over state-of-the-art accelerators.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 67 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

43 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025WaferLLM: Large Language Model Inference at Wafer Scale6 Feb
  2. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
29 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  7. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  10. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  11. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  12. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  13. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  14. 2024Residual Quantization with Implicit Neural Codebookscited by 25
  15. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 21
  16. 2025HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understandingcited by 19
  17. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 18
  18. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
  19. 2024CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activationcited by 14
  20. 2025GenAI at the Edge: Comprehensive Survey on Empowering Edge Devicescited by 14
  21. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 14
  22. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  23. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 8
  24. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  25. 2025Transitive Array: An Efficient GEMM Accelerator with Result Reusecited by 7
  26. 2025Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networkscited by 5
  27. 2023Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Acceleratorscited by 4
  28. 2025Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplicationcited by 2
  29. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Speed Odyssey for Deployable Quantization of LLMs
  2. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  3. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
  4. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  5. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple
  6. 2026LLM Inference in a Flash!
  7. 2024Communication Compression for Tensor Parallel LLM Inference
  8. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design