TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR
Not on the map. Why?
Abstract (arXiv)
In traditional conversational intelligence from speech, a cascaded pipeline is used, involving tasks such as voice activity detection, diarization, transcription, and subsequent processing with different NLP models for tasks like semantic endpointing and named entity recognition (NER). Our paper introduces TokenVerse, a single Transducer-based model designed to handle multiple tasks. This is achieved by integrating task-specific tokens into the reference text during ASR model training, streamlining the inference and eliminating the need for separate NLP models. In addition to ASR, we conduct experiments on 3 different tasks: speaker change detection, endpointing, and NER. Our experiments on a public and a private dataset show that the proposed method improves ASR by up to 7.7% in relative WER while outperforming the cascaded pipeline approach in individual task performance. Our code is publicly available: https://github.com/idiap/tokenverse-unifying-speech-nlp
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 5 Jul 2024.
Builds on
25 of 36 references.
References · 25
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
- 2018A Survey on Deep Learning for Named Entity Recognitioncited by 281
- 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
- 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 109
- 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 67
- 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
13 more
- 2020Topic-Aware Multi-turn Dialogue Modelingcited by 46
- 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
- 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
- 2020End-to-end Named Entity Recognition from English Speechcited by 36
- 2021Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detectioncited by 31
- 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
- 2020Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modelingcited by 23
- 2019Audio De-identification: A New Entity Recognition Taskcited by 14
- 2018End-to-end named entity extraction from speechcited by 11
- 2021Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuningcited by 4
- 2023t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capabilitycited by 4
- 2024SpeechComposer: Unifying Multiple Speech Tasks with Prompt Compositioncited by 3
- 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.