# Knowledge boosting during low-latency inference

arXiv: https://arxiv.org/abs/2407.11055 · HTML: https://mlsanity.com/paper/2407.11055/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [CoEdge: Cooperative DNN Inference with Adaptive Workload Partitioning over Heterogeneous Edge Devices](https://mlsanity.com/paper/2012.03257/) (2020)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [WHAMR!: Noisy and Reverberant Single-Channel Speech Separation](https://mlsanity.com/paper/1910.10279/) (2019)
- [Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy](https://mlsanity.com/paper/1806.07840/) (2018)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation](https://mlsanity.com/paper/2002.12597/) (2020)
- [Look Once to Hear: Target Speech Hearing with Noisy Examples](https://mlsanity.com/paper/2405.06289/) (2024)
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [Sub-Band Knowledge Distillation Framework for Speech Enhancement](https://mlsanity.com/paper/2005.14435/) (2020)
- [ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement](https://mlsanity.com/paper/2206.13611/) (2022)
- [Low bit rate binaural link for improved ultra low-latency low-complexity multichannel speech enhancement in Hearing Aids](https://mlsanity.com/paper/2307.08858/) (2023)
- [Unifying Robustness and Fidelity: A Comprehensive Study of Pretrained Generative Methods for Speech Enhancement in Adverse Conditions](https://mlsanity.com/paper/2309.09028/) (2023)

## Built on by (most cited first, 4 of 4)

- [TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers](https://mlsanity.com/paper/2412.15304/) (2024)
- [Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement](https://mlsanity.com/paper/2411.02019/) (2024)
- [DeDelayed: Deleting Remote Inference Delay via On-Device Correction](https://mlsanity.com/paper/2510.13714/) (2025)
- [Cloud-Boosted Low-Compute Multi-Channel Speech Enhancement](https://mlsanity.com/paper/2608.07423/) (2026)

## Abstract (arXiv)

Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A possible solution is to transfer hints during inference from a large model running remotely to a small model running on-device. However, this incurs a communication delay that breaks real-time requirements and does not guarantee that both models will operate on the same data at the same time. We propose knowledge boosting, a novel technique that allows a large model to operate on time-delayed input during inference, while still boosting small model performance. Using a streaming neural network that processes 8 ms chunks, we evaluate different speech separation and enhancement tasks with communication delays of up to six chunks or 48 ms. Our results show larger gains where the performance gap between the small and large models is wide, demonstrating a promising method for large-small model collaboration for low-latency applications. Code, dataset, and audio samples available at https://knowledgeboosting.cs.washington.edu/.