# Topic Identification For Spontaneous Speech: Enriching Audio Features With Embedded Linguistic Information

arXiv: https://arxiv.org/abs/2307.11450 · HTML: https://mlsanity.com/paper/2307.11450/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [Audio-Linguistic Embeddings for Spoken Sentences](https://mlsanity.com/paper/1902.07817/) (2019)
- [Lahjoita puhetta -- a large-scale corpus of spoken Finnish with some benchmarks](https://mlsanity.com/paper/2203.12906/) (2022)
- [Topic Identification for Speech without ASR](https://mlsanity.com/paper/1703.07476/) (2017)
- [Topic Spotting using Hierarchical Networks with Self Attention](https://mlsanity.com/paper/1904.02815/) (2019)

## Abstract (arXiv)

Traditional topic identification solutions from audio rely on an automatic speech recognition system (ASR) to produce transcripts used as input to a text-based model. These approaches work well in high-resource scenarios, where there are sufficient data to train both components of the pipeline. However, in low-resource situations, the ASR system, even if available, produces low-quality transcripts, leading to a bad text-based classifier. Moreover, spontaneous speech containing hesitations can further degrade the performance of the ASR model. In this paper, we investigate alternatives to the standard text-only solutions by comparing audio-only and hybrid techniques of jointly utilising text and audio features. The models evaluated on spontaneous Finnish speech demonstrate that purely audio-based solutions are a viable option when ASR components are not available, while the hybrid multi-modal solutions achieve the best results.