mlsanityData to 14 Oct 2026

UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge

arXiv 2608.09291 ↗First posted 10 Aug 2026cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

27 of 46 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
15 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2020Sparse GPU Kernels for Deep Learningcited by 141
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  7. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  8. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 34
  9. 2024High Performance Unstructured SpMM Computation Using Tensor Corescited by 8
  10. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edgecited by 6
  11. 2026Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEscited by 5
  12. 2025Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Corescited by 4
  13. 2023Intelligent Assistant Language Understanding On Devicecited by 3
  14. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devicescited by 2
  15. 2026HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission
  2. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  3. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  4. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  5. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference
  6. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
  7. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
  8. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference