# Speech Understanding on Tiny Devices with A Learning Cache

arXiv: https://arxiv.org/abs/2311.18188 · HTML: https://mlsanity.com/paper/2311.18188/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 69 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [A Survey on Spoken Language Understanding: Recent Advances and New Frontiers](https://mlsanity.com/paper/2103.03095/) (2021)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1703.05390/) (2017)
- [Personalized Speech recognition on mobile devices](https://mlsanity.com/paper/1603.03185/) (2016)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding](https://mlsanity.com/paper/2305.13512/) (2023)
- [Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions](https://mlsanity.com/paper/1911.02086/) (2019)
- [Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks](https://mlsanity.com/paper/1609.01462/) (2016)
- [Query-by-example on-device keyword spotting](https://mlsanity.com/paper/1910.05171/) (2019)
- [Improving End-to-End Speech-to-Intent Classification with Reptile](https://mlsanity.com/paper/2008.01994/) (2020)
- [DONUT: CTC-based Query-by-Example Keyword Spotting](https://mlsanity.com/paper/1811.10736/) (2018)
- [Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning](https://mlsanity.com/paper/2007.01836/) (2020)
- [End-to-end architectures for ASR-free spoken language understanding](https://mlsanity.com/paper/1910.10599/) (2019)
- [Encoder-decoder with Focus-mechanism for Sequence Labelling Based Spoken Language Understanding](https://mlsanity.com/paper/1608.02097/) (2016)
- [A Streaming End-to-End Framework For Spoken Language Understanding](https://mlsanity.com/paper/2105.10042/) (2021)
- [FANS: Fusing ASR and NLU for on-device SLU](https://mlsanity.com/paper/2111.00400/) (2021)

## Built on by (most cited first, 3 of 3)

- [DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Training](https://mlsanity.com/paper/2507.07149/) (2025)
- [QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding](https://mlsanity.com/paper/2505.14723/) (2025)
- [Enabling Automatic Self-Talk Detection via Earables](https://mlsanity.com/paper/2511.07493/) (2025)

## Abstract (arXiv)

This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to any cached ones to the cloud for full inference. Realization of this idea, however, is non-trivial: the device needs to compare acoustic features in a robust yet low-cost way. To this end, we present SpeechCache (or SC), a speech cache for tiny devices. It matches speech inputs at two levels of representations: first by sequences of clustered raw sound units, then as sequences of phonemes. Working in tandem, the two representations offer complementary tradeoffs between cost and efficiency. To boost accuracy even further, our cache learns to personalize: with the mismatched and then offloaded inputs, it continuously finetunes the device's feature extractors with the assistance of the cloud. We implement SC on an off-the-shelf STM32 microcontroller. The complete implementation has a small memory footprint of 2MB. Evaluated on challenging speech benchmarks, our system resolves 45%-90% of inputs on device, reducing the average latency by up to 80% compared to offloading to popular cloud speech recognition services. The benefit brought by our proposed SC is notable even in adversarial settings - noisy environments, cold cache, or one device shared by a number of users.