mlsanityData to 13 Oct 2026

GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

arXiv 2005.03842 ↗First posted 8 May 2020cs.LG cs.AR stat.MLcited by 106

Not on the map. Why?

Abstract (arXiv)

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.

Key facts. Cited by 106 papers in the map (data through 13 Oct 2026). First posted on arXiv 8 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 14; 2022: 24; 2023: 21; 2024: 24; 2025: 16; 2026: 7.

106 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 569
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 322
  5. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 319
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 292
  7. 2021Post-Training Quantization for Vision Transformercited by 278
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 254
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 204
  10. 2023Speculative Decoding with Big Little Decodercited by 162
  11. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 150
38 more
  1. 2021Learned Token Pruning for Transformerscited by 139
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  4. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  5. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  6. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  7. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  8. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  9. 2022BiBERT: Accurate Fully Binarized BERTcited by 72
  10. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 68
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  12. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  13. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  14. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  15. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  16. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  17. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  18. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  19. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34
  20. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  21. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  22. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  23. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  24. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  25. 2021Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attentioncited by 26
  26. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  27. 2023X-Former: In-Memory Acceleration of Transformerscited by 25
  28. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  29. 2022EcoFormer: Energy-Saving Attention with Linear Complexitycited by 21
  30. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 19
  31. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  32. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 18
  33. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 18
  34. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
  35. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantizationcited by 15
  36. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  37. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  38. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14

50 most cited of 106 All 106 (JSON)