mlsanityData to 7 Oct 2026

The Case for Co-Designing Model Architectures with Hardware

arXiv 2401.14489 ↗First posted 25 Jan 2024cs.DC cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new deep learning (DL) models. As a consequence, modifying a DL model to be more amenable to the target hardware can significantly improve the runtime performance of DL training and inference. In this paper, we provide a set of guidelines for users to maximize the runtime performance of their transformer models. These guidelines have been created by carefully considering the impact of various model hyperparameters controlling model shape on the efficiency of the underlying computation kernels executed on the GPU. We find the throughput of models with efficient model shapes is up to 39% higher while preserving accuracy compared to models with a similar number of parameters but with unoptimized shapes.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
11 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  6. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  7. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  8. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  9. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  10. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  11. 2018Modeling Deep Learning Accelerator Enabled GPUscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 2.

11 papers.

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2024The Zamba2 Suite: Technical Reportcited by 20
  3. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  4. 2024A Scalable Real-Time Data Assimilation Framework for Predicting Turbulent Atmosphere Dynamicscited by 5
  5. 2024Best Practices for Large Language Models in Radiologycited by 5
  6. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
  7. 2024Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inferencecited by 4
  8. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 4
  9. 2025Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequencescited by 3
  10. 2025Chinese ModernBERT with Whole-Word Maskingcited by 0
  11. 2026NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Modelcited by 0