mlsanityData to 14 Oct 2026

Mugi: Value Level Parallelism For Efficient LLMs

arXiv 2601.10823 ↗First posted 15 Jan 2026cs.LG cs.ARcited by 0

Not on the map. Why?

Abstract (arXiv)

Value level parallelism (VLP) has been proposed to improve the efficiency of large-batch, low-precision general matrix multiply (GEMM) between symmetric activations and weights. In transformer based large language models (LLMs), there exist more sophisticated operations beyond activation-weight GEMM. In this paper, we explore how VLP benefits LLMs. First, we generalize VLP for nonlinear approximations, outperforming existing nonlinear approximations in end-to-end LLM accuracy, performance, and efficiency. Our VLP approximation follows a value-centric approach, where important values are assigned with greater accuracy. Second, we optimize VLP for small-batch GEMMs with asymmetric inputs efficiently, which leverages timely LLM optimizations, including weight-only quantization, key-value (KV) cache quantization, and group query attention. Finally, we design a new VLP architecture, Mugi, to encapsulate the innovations above and support full LLM workloads, while providing better performance, efficiency and sustainability. Our experimental results show that Mugi can offer significant improvements on throughput and energy efficiency, up to 45× and 668× for nonlinear softmax operations, and 2.07× and 3.11× for LLMs, and also decrease operational carbon for LLM operation by 1.45× and embodied carbon by 1.48×.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 87 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

35 of 87 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2019Searching for MobileNetV3cited by 1,994
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
23 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  11. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  12. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  13. 2022FP8 Formats for Deep Learningcited by 196
  14. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  15. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  16. 2022FP8 Quantization: The Power of the Exponentcited by 88
  17. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  18. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
  19. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  20. 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 22
  21. 2018C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAscited by 21
  22. 2024SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processorscited by 4
  23. 2021Representation range needs for 16-bit neural network trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  2. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  3. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
  4. 2025LLM Inference Acceleration via Efficient Operation Fusion
  5. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
  6. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  7. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  8. 2023Efficient LLM Inference on CPUs