# Real-time Speech Interruption Analysis: From Cloud to Client Deployment

arXiv: https://arxiv.org/abs/2210.13334 · HTML: https://mlsanity.com/paper/2210.13334/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation](https://mlsanity.com/paper/2202.07959/) (2022)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [FastAudio: A Learnable Audio Front-End for Spoof Speech Detection](https://mlsanity.com/paper/2109.02774/) (2021)
- [On training targets for noise-robust voice activity detection](https://mlsanity.com/paper/2102.07445/) (2021)
- [Meeting Effectiveness and Inclusiveness in Remote Collaboration](https://mlsanity.com/paper/2102.09803/) (2021)
- [Ultra Fast Speech Separation Model with Teacher Student Learning](https://mlsanity.com/paper/2204.12777/) (2022)

## Abstract (arXiv)

Meetings are an essential form of communication for all types of organizations, and remote collaboration systems have been much more widely used since the COVID-19 pandemic. One major issue with remote meetings is that it is challenging for remote participants to interrupt and speak. We have recently developed the first speech interruption analysis model, which detects failed speech interruptions, shows very promising performance, and is being deployed in the cloud. To deliver this feature in a more cost-efficient and environment-friendly way, we reduced the model complexity and size to ship the WavLM_SI model in client devices. In this paper, we first describe how we successfully improved the True Positive Rate (TPR) at a 1% False Positive Rate (FPR) from 50.9% to 68.3% for the failed speech interruption detection model by training on a larger dataset and fine-tuning. We then shrank the model size from 222.7 MB to 9.3 MB with an acceptable loss in accuracy and reduced the complexity from 31.2 GMACS (Giga Multiply-Accumulate Operations per Second) to 4.3 GMACS. We also estimated the environmental impact of the complexity reduction, which can be used as a general guideline for large Transformer-based models, and thus make those models more accessible with less computation overhead.