mlsanityData to 7 Oct 2026

A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration

arXiv 2111.15024 ↗First posted 29 Nov 2021cs.AR cs.LGcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

This work focuses on an efficient Agile design methodology for domain-specific accelerators. We employ feature-by-feature enhancement of a vertical development stack and apply it to the TVM/VTA inference accelerator. We have enhanced the VTA design space and enabled end-to-end support for additional workloads. This has been accomplished by augmenting the VTA micro-architecture and instruction set architecture (ISA), as well as by enhancing the TVM compilation stack to support a wide range of VTA configs. The VTA tsim implementation (CHISEL-based) has been enhanced with fully pipelined versions of the ALU/GEMM execution units. In tsim, memory width can now range between 8-64 bytes. Field widths have been made more flexible to support larger scratchpads. New instructions have been added: element-wise 8-bit multiplication to support depthwise convolution, and load with a choice of pad values to support max pooling. Support for more layers and better double buffering has also been added. Fully pipelining ALU/GEMM helps significantly: 4.9x fewer cycles with minimal area change to run ResNet-18 under the default config. Configs featuring a further 11.5x decrease in cycle count at a cost of 12x greater area can be instantiated. Many points on the area-performance pareto curve are shown, showcasing the balance of execution unit sizing, memory interface width, and scratchpad sizing. Finally, VTA is now able to run Mobilenet 1.0 and all layers for ResNets, including the previously disabled pooling and fully connected layers. The TVM/VTA architecture has always featured end-to-end workload evaluation on RTL in minutes. With our modifications, it now offers a much greater number of feasible configurations with a wide range of cost vs. performance. All capabilities mentioned are available in opensource forks while a subset of these capabilities have already been upstreamed.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 44 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

18 of 44 references.

References · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  5. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 199
  6. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  7. 2018Learning to Optimize Tensor Programscited by 138
  8. 2018Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractionscited by 84
  9. 2020MLIR: A Compiler Infrastructure for the End of Moore's Lawcited by 64
  10. 2018TVM: An Automated End-to-End Optimizing Compiler for Deep Learningcited by 62
  11. 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
  12. 2019NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learningcited by 37
6 more
  1. 2018Optimizing CNN Model Inference on CPUscited by 29
  2. 2018Relay: A New IR for Machine Learning Frameworkscited by 24
  3. 2020AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learningcited by 24
  4. 2020FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloadscited by 9
  5. 2018A Hardware-Software Blueprint for Flexible Deep Learning Specializationcited by 7
  6. 2021RHNAS: Realizable Hardware and Neural Architecture Searchcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2024Dynamic Co-Optimization Compiler: Leveraging Multi-Agent Reinforcement Learning for Enhanced DNN Accelerator Performancecited by 4
  2. 2023An Open-Source ML-Based Full-Stack Optimization Framework for Machine Learning Acceleratorscited by 2
  3. 2026MAGE: Human-Like Macro Placement via Agentic Multimodal Reasoningcited by 1
  4. 2023Performance Analysis of DNN Inference/Training with Convolution and non-Convolution Operationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Scalable FPGA Architecture With Adaptive Memory Utilization for GEMM-Based Operations
  2. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
  3. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  4. 2025Flexible Vector Integration in Embedded RISC-V SoCs for End to End CNN Inference Acceleration ↗
  5. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
  6. 2026ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA
  7. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference
  8. 2025FlexiSAGA: A Flexible Systolic Array GEMM Accelerator for Sparse and Dense Processing