# Architectural Implications of Neural Network Inference for High Data-Rate, Low-Latency Scientific Applications

arXiv: https://arxiv.org/abs/2403.08980 · HTML: https://mlsanity.com/paper/2403.08980/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [LogicNets: Co-Designed Neural Networks and Circuits for Extreme-Throughput Applications](https://mlsanity.com/paper/2004.03021/) (2020)
- [A reconfigurable neural network ASIC for detector front-end data compression at the HL-LHC](https://mlsanity.com/paper/2105.01683/) (2021)
- [FastML Science Benchmarks: Accelerating Real-Time Scientific Edge Machine Learning](https://mlsanity.com/paper/2207.07958/) (2022)
- [Parallel Programming for FPGAs](https://mlsanity.com/paper/1805.03648/) (2018)
- [Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark](https://mlsanity.com/paper/2206.11791/) (2022)
- [Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak](https://mlsanity.com/paper/2312.00128/) (2023)

## Built on by (most cited first, 2 of 2)

- [Reliable edge machine learning hardware for scientific applications](https://mlsanity.com/paper/2406.19522/) (2024)
- [Design Rules for Extreme-Edge Scientific Computing on AI Engines](https://mlsanity.com/paper/2604.19106/) (2026)

## Abstract (arXiv)

With more scientific fields relying on neural networks (NNs) to process data incoming at extreme throughputs and latencies, it is crucial to develop NNs with all their parameters stored on-chip. In many of these applications, there is not enough time to go off-chip and retrieve weights. Even more so, off-chip memory such as DRAM does not have the bandwidth required to process these NNs as fast as the data is being produced (e.g., every 25 ns). As such, these extreme latency and bandwidth requirements have architectural implications for the hardware intended to run these NNs: 1) all NN parameters must fit on-chip, and 2) codesigning custom/reconfigurable logic is often required to meet these latency and bandwidth constraints. In our work, we show that many scientific NN applications must run fully on chip, in the extreme case requiring a custom chip to meet such stringent constraints.