mlsanityData to 14 Oct 2026

A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators

arXiv 2105.12842 ↗First posted 26 May 2021cs.LG cs.AR cs.PFcited by 19

Not on the map. Why?

Abstract (arXiv)

The rapidly-changing deep learning landscape presents a unique opportunity for building inference accelerators optimized for specific datacenter-scale workloads. We propose Full-stack Accelerator Search Technique (FAST), a hardware accelerator search framework that defines a broad optimization environment covering key design decisions within the hardware-software stack, including hardware datapath, software scheduling, and compiler passes such as operation fusion and tensor padding. In this paper, we analyze bottlenecks in state-of-the-art vision and natural language processing (NLP) models, including EfficientNet and BERT, and use FAST to design accelerators capable of addressing these bottlenecks. FAST-generated accelerators optimized for single workloads improve Perf/TDP by 3.7x on average across all benchmarks compared to TPU-v3. A FAST-generated accelerator optimized for serving a suite of workloads improves Perf/TDP by 2.4x on average compared to TPU-v3. Our return on investment analysis shows that FAST-generated accelerators can potentially be practical for moderate-sized datacenter deployments.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 45 of its 123 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

45 of 123 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
33 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  4. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  5. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  6. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  7. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  8. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  9. 2018Online normalizer calculation for softmaxcited by 162
  10. 2014Bayesian Optimization with Unknown Constraintscited by 109
  11. 2015Fast Algorithms for Convolutional Neural Networkscited by 102
  12. 2019Towards Unconstrained End-to-End Text Spottingcited by 57
  13. 2019Hardware/Software Co-Exploration of Neural Architecturescited by 50
  14. 2019FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edgecited by 48
  15. 2020Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Taskscited by 43
  16. 2021An Evaluation of Edge TPU Accelerators for Convolutional Neural Networkscited by 39
  17. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 36
  18. 2020EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutionscited by 36
  19. 2020ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learningcited by 35
  20. 2018Interstellar: Using Halide's Scheduling Language to Analyze DNN Acceleratorscited by 34
  21. 2020Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Acceleratorcited by 30
  22. 2020Transferable Graph Optimizers for ML Compilerscited by 29
  23. 2021Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Searchcited by 24
  24. 2021Searching for Fast Model Families on Datacenter Acceleratorscited by 22
  25. 2020DNNExplorer: A Framework for Modeling and Exploring a Novel Paradigm of FPGA-based DNN Acceleratorcited by 18
  26. 2021Apollo: Transferable Architecture Explorationcited by 17
  27. 2018On Optimizing Operator Fusion Plans for Large-Scale Machine Learning in SystemMLcited by 13
  28. 2021Rethinking Co-design of Neural Architectures and Hardware Acceleratorscited by 12
  29. 2020Learned Hardware/Software Co-Design of Neural Acceleratorscited by 9
  30. 2020A Learned Performance Model for Tensor Processing Unitscited by 2
  31. 2013A cost-benefit analysis of low latency via added utilizationcited by 1
  32. 2017Ascertaining Uncertainty for Efficient Exact Cache Analysiscited by 1
  33. 2018FusionStitching: Deep Fusion and Code Generation for Tensorflow Computations on GPUscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 8; 2024: 2; 2025: 4; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  2. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  3. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 19
  4. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 18
  5. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  6. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  7. 2025Autocomp: A Powerful and Portable Code Optimizer for Tensor Acceleratorscited by 13
  8. 2023Training Large Language Models Efficiently with Sparsity and Dataflowcited by 7
  9. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  10. 2022Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systemscited by 6
  11. 2023ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Designcited by 6
  12. 2023Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chipscited by 5
7 more
  1. 2025DOSA: Differentiable Model-Based One-Loop Search for DNN Acceleratorscited by 4
  2. 2024AI-Guided Codesign Framework for Novel Material and Device Design applied to MTJ-based True Random Number Generatorscited by 1
  3. 2026AI+HW 2035: Shaping the Next Decadecited by 1
  4. 2022Energy-efficient DNN Inference on Approximate Accelerators Through Formal Property Explorationcited by 0
  5. 2022Multi-Agent Reinforcement Learning for Microprocessor Design Space Explorationcited by 0
  6. 2023AutoML for neuromorphic computing and application-driven co-design: asynchronous, massively parallel optimization of spiking architecturescited by 0
  7. 2023NicePIM: Design Space Exploration for Processing-In-Memory DNN Accelerators with 3D-Stacked-DRAMcited by 0