mlsanityData to 7 Oct 2026

A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

arXiv 2202.08509 ↗First posted 17 Feb 2022cs.SD cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing a compact audio-visual WWS system by utilizing visual information to alleviate the degradation. Specifically, in order to use visual information, we first encode the detected lips to fixed-size vectors with MobileNet and concatenate them with acoustic features followed by the fusion network for WWS. However, the audio-visual model based on neural networks requires a large footprint and a high computational complexity. To meet the application requirements, we introduce a neural network pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning manner (LTH-IF), to the single-modal and multi-modal models, respectively. Tested on our in-house corpus for audio-visual WWS in a home TV scene, the proposed audio-visual system achieves significant performance improvements over the single-modality (audio-only or video-only) system under different noisy conditions. Moreover, LTH-IF pruning can largely reduce the network parameters and computations with no degradation of WWS performance, leading to a potential product solution for the TV wake-up scenario.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

22 of 31 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  5. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  6. 2018Federated Learning for Keyword Spottingcited by 95
  7. 2017Deep Residual Learning for Small-Footprint Keyword Spottingcited by 68
  8. 2020Streaming keyword spotting on mobile devicescited by 47
  9. 2017Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spottingcited by 30
  10. 2020Accurate Detection of Wake Word Start and End Using a CNNcited by 24
  11. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  12. 2020Seeing wake words: Audio-visual Keyword Spottingcited by 22
10 more
  1. 2019Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutionscited by 20
  2. 2020Depthwise Separable Convolutional ResNet with Squeeze-and-Excitation Blocks for Small-footprint Keyword Spottingcited by 15
  3. 2020Small-Footprint Keyword Spotting with Multi-Scale Temporal Convolutioncited by 15
  4. 2020Towards Data-efficient Modeling for Wake Word Spottingcited by 13
  5. 2019Predicting detection filters for small footprint open-vocabulary keyword spottingcited by 10
  6. 2020AutoKWS: Keyword Spotting with Differentiable Architecture Searchcited by 10
  7. 2021Wake Word Detection with Streaming Transformerscited by 10
  8. 2020Multi-Task Network for Noise-Robust Keyword Spotting and Speaker Verification using CTC-based Soft VAD and Global Query Attentioncited by 9
  9. 2020Optimize what matters: Training DNN-HMM Keyword Spotting Model Using End Metriccited by 9
  10. 2021A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classificationcited by 4

Built on by

Not cited yet.