# Empowering In-Browser Deep Learning Inference on Edge Devices with Just-in-Time Kernel Optimizations

arXiv: https://arxiv.org/abs/2309.08978 · HTML: https://mlsanity.com/paper/2309.08978/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 6 of 6)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation](https://mlsanity.com/paper/2001.08743/) (2020)
- [Moving Deep Learning into Web Browser: How Far Can We Go?](https://mlsanity.com/paper/1901.09388/) (2019)

## Built on by (most cited first, 4 of 4)

- [Anatomizing Deep Learning Inference in Web Browsers](https://mlsanity.com/paper/2402.05981/) (2024)
- [[Emerging Ideas] Artificial Tripartite Intelligence: A Bio-Inspired, Sensor-First Architecture for Physical AI](https://mlsanity.com/paper/2604.13959/) (2026)
- [GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEs](https://mlsanity.com/paper/2609.35882/) (2026)
- [GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference](https://mlsanity.com/paper/2610.17321/) (2026)

## Abstract (arXiv)

Web is increasingly becoming the primary platform to deliver AI services onto edge devices, making in-browser deep learning (DL) inference more prominent. Nevertheless, the heterogeneity of edge devices, combined with the underdeveloped state of Web hardware acceleration practices, hinders current in-browser inference from achieving its full performance potential on target devices. To address this issue, this paper presents the pioneering inbrowser inference system, nnJIT, which enables just-in-time (JIT) auto-generation of optimized computing kernels for edge devices. nnJIT is built upon two novel techniques that significantly reduce kernel search and compilation overhead while improving performance firmly: Tensor-Web Compiling Co-Design lowers compiling costs by around 100X through eliminating redundant and ineffective compiling passes; Web-Specific Lite Kernel Optimization Space reduces kernel tuning costs by focusing on Web programming requirements and efficient device resource utilization, pruning the optimization space from millions to only dozens. nnJIT is evaluated for modern models, e.g., BART, T5, and Llama 2, on a range of edge devices including laptops and smartphones using different browsers and hardware from ARM, Intel, AMD and Nvidia. The results show that nnJIT can achieve up to 8.2X faster within 30 seconds compared to the existing baselines.