mlsanityData to 14 Oct 2026

SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version

arXiv 2508.15478 ↗First posted 21 Aug 2025cs.CL cs.CY cs.PFcited by 3

Not on the map. Why?

Abstract (arXiv)

Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 67 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

36 of 67 references.

References · 36

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
24 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  5. 2022Teaching Small Language Models to Reasoncited by 284
  6. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  7. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  8. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  9. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  10. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  11. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  12. 2019Energy Usage Reports: Environmental awareness as part of algorithmic accountabilitycited by 44
  13. 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
  14. 2018Findings of the E2E NLG Challengecited by 28
  15. 2024Large Language Models for Manufacturingcited by 24
  16. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  17. 2023Watt For What: Rethinking Deep Learning's Energy-Performance Relationshipcited by 16
  18. 2024Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Trainingcited by 15
  19. 2024Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Financecited by 14
  20. 2023Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inferencecited by 13
  21. 2023Energy and Carbon Considerations of Fine-Tuning BERTcited by 11
  22. 2019Long-length Legal Document Classificationcited by 10
  23. 2022Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Modelscited by 10
  24. 2024Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classificationcited by 1
  2. 2026Little Brains, Big Feats: Exploring Compact Language Modelscited by 0
  3. 2026RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistantscited by 0