mlsanityData to 7 Oct 2026

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

arXiv 2405.10299 ↗First posted 16 May 2024cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consumption, GPU memory usage, and performance. Identifying optimal model configurations under specific hardware constraints is becoming essential but remains challenging due to the computational load of exhaustive training and evaluation on multiple devices. To address this, we introduce HW-GPT-Bench, a hardware-aware benchmark that utilizes surrogate predictions to approximate various hardware metrics across 13 devices of architectures in the GPT-2 family, with architectures containing up to 1.55B parameters. Our surrogates, via calibrated predictions and reliable uncertainty estimates, faithfully model the heteroscedastic noise inherent in the energy and latency measurements. To estimate perplexity, we employ weight-sharing techniques from Neural Architecture Search (NAS), inheriting pretrained weights from the largest GPT-2 model. Finally, we demonstrate the utility of HW-GPT-Bench by simulating optimization trajectories of various multi-objective optimization algorithms in just a few seconds.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 72 of its 103 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

72 of 103 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  10. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  11. 2020Designing Network Design Spacescited by 954
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
38 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  5. 2019Conformalized Quantile Regressioncited by 620
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  8. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  10. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  11. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  12. 2018Slimmable Neural Networkscited by 301
  13. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  14. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  15. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  16. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  17. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  18. 2019Structured Pruning of Large Language Modelscited by 201
  19. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  20. 2023Efficient Large Language Models: A Surveycited by 183
  21. 2019The Evolved Transformercited by 180
  22. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  23. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  24. 2021End-to-End Constrained Optimization Learning: A Surveycited by 154
  25. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  26. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  27. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  28. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  29. 2018A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizationscited by 98
  30. 2018Searching for Efficient Multi-Scale Architectures for Dense Image Predictioncited by 96
  31. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  32. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  33. 2021HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmarkcited by 83
  34. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  35. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  36. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
  37. 2020NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Searchcited by 75
  38. 2019NAS evaluation is frustratingly hardcited by 69

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2026AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensivecited by 4
  2. 2025Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankerscited by 3
  3. 2025Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillationcited by 1
  4. 2024NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performancecited by 0
  5. 2025Optimizing edge AI models on HPC systems with the edge in the loopcited by 0
  6. 2026LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Modelscited by 0
  7. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  8. 2026Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agentscited by 0
  9. 2026Rank Portability Does Not Imply Feasibility Portability: Target-Specific Evaluation of Joint Hardware Constraintscited by 0
  10. 2026ScAn-Bench: Evaluating Scaling Analysis Methodologycited by 0