mlsanityData to 7 Oct 2026

An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2

arXiv 2411.12758 ↗First posted 15 Nov 2024cs.CL cs.AI cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

22 of 39 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  11. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
10 more
  1. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  4. 2023A Systematic Review of Green AIcited by 77
  5. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  8. 2021Smart at what cost? Characterising Mobile Deep Neural Networks in the wildcited by 25
  9. 2020Towards Accurate and Reliable Energy Measurement of NLP Modelscited by 16
  10. 2023Can pruning make Large Language Models more efficient?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  2. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspectivecited by 8
  3. 2025Green MLOps to Green GenOps: An Empirical Study of Energy Consumption in Discriminative and Generative AI Operationscited by 2
  4. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0