mlsanityData to 7 Oct 2026

The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment

arXiv 2302.06117 ↗First posted 13 Feb 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Increased focus on the computational efficiency of NLP systems has motivated the design of efficient model architectures and improvements to underlying hardware accelerators. However, the resulting increases in computational throughput and reductions in floating point operations have not directly translated to improvements in wall-clock inference latency. We demonstrate that these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomenon as the textitframework tax, and observe that the disparity is growing as hardware speed increases over time. In this work, we examine this phenomenon through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency. Code is available at https://github.com/JaredFern/Framework-Tax.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

52 of 64 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016Wide Residual Networkscited by 3,057
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2020Reformer: The Efficient Transformercited by 1,658
38 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  6. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  7. 2021Going deeper with Image Transformerscited by 759
  8. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  9. 2019Green AIcited by 620
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  13. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  14. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  15. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  16. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  17. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  18. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  19. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  20. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  21. 2019MLPerf Inference Benchmarkcited by 195
  22. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  23. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  24. 2020The Hardware Lotterycited by 160
  25. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  26. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  27. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  28. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  29. 2021The Efficiency Misnomercited by 98
  30. 2021How to Train BERT with an Academic Budgetcited by 80
  31. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64
  32. 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
  33. 2022Flashlight: Enabling Innovation in Tools for Machine Learningcited by 25
  34. 2020Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Trainingcited by 22
  35. 2020HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processingcited by 16
  36. 2017Deep Learning with Dynamic Computation Graphscited by 15
  37. 2020MLPerf Mobile Inference Benchmarkcited by 14
  38. 2022Building a Performance Model for Deep Learning Recommendation Model Training on GPUscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 1; 2026: 4.

11 papers.

  1. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insightscited by 65
  2. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edgecited by 8
  3. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  4. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  5. 2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
  6. 2024Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generationcited by 7
  7. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  8. 2026Shallow-π: Knowledge Distillation for Flow-based VLAscited by 7
  9. 2023Spatial-Language Attention Policies for Efficient Robot Learningcited by 3
  10. 2026TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decompositioncited by 2
  11. 2026Evaluation of ML Resource Utilization Requires Model Life Cycle Assessmentcited by 0