mlsanityData to 14 Oct 2026

Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models

arXiv 2604.25903 ↗First posted 28 Apr 2026cs.SE cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The accelerating adoption of Large Language Models (LLMs) in software engineering (SE) has brought with it a silent crisis: unsustainable computational cost. While these models demonstrate remarkable capabilities in different SE tasks, they are unmanageably large, slow to deploy, memory-intensive, and carbon-heavy. This reality threatens not only the scalability and accessibility of AI-powered SE, but also its long-term environmental sustainability. The research challenge is clear: we must go beyond accuracy and address efficiency and environmental cost as first-class design constraints. To meet this challenge, we introduce Carbon-Taxed Transformers (CTT), a systematic multi-architectural compression principled pipeline ordering inspired by economic carbon taxation principles. Drawing from the economic concept of carbon pricing, CTT operationalizes a computational carbon tax that penalizes architectural inefficiencies and rewards deployment-ready compression. We evaluate CTT across three core SE tasks: code clone detection, code summarization, and code generation, with models spanning encoder-only, encoder-decoder, and decoder-only architecture. Our results show that CTT delivers on inference: (1) up to 49x memory reduction, (2) time reduction up to 8-10x for clone detection, up to 3x for summarization, and 4-7x for generation, (3) up to 81% reduction in CO2 emissions and (4) CTT retains around 98% accuracy on clone detection, around 89% on summarization, and up to 91% (textual metrics) and 68% (pass@1) for generation. Two ablation studies show that pipeline ordering and individual component contributions are both essential, providing empirical justification for CTT's design and effectiveness. This work establishes a viable path toward responsible AI in SE through aggressive yet performance-preserving compression.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

54 of 72 references.

References · 54

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
38 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  10. 2019Are Sixteen Heads Really Better than One?cited by 816
  11. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  14. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  15. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  18. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  19. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  20. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  21. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  22. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  23. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  24. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  25. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  26. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  27. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  28. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  29. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  30. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 166
  31. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  32. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  33. 2021Learned Token Pruning for Transformerscited by 139
  34. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  35. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  36. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  37. 2023Distilled GPT for Source Code Summarizationcited by 31
  38. 2022Compressing Pre-trained Models of Code into 3 MBcited by 21

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.