# TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation

arXiv: https://arxiv.org/abs/2508.19856 · HTML: https://mlsanity.com/paper/2508.19856/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [Topic-Aware Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2009.12539/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [End-to-end Named Entity Recognition from English Speech](https://mlsanity.com/paper/2005.11184/) (2020)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling](https://mlsanity.com/paper/2012.07311/) (2020)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward](https://mlsanity.com/paper/2411.03866/) (2024)
- [Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study](https://mlsanity.com/paper/2401.12789/) (2024)
- [XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models](https://mlsanity.com/paper/2407.04439/) (2024)
- [SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition](https://mlsanity.com/paper/2401.18045/) (2024)
- [TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR](https://mlsanity.com/paper/2407.04444/) (2024)

## Abstract (arXiv)

Token-based multitasking frameworks like TokenVerse require all training utterances to have labels for all tasks, hindering their ability to leverage partially annotated datasets and scale effectively. We propose TokenVerse++, which introduces learnable vectors in the acoustic embedding space of the XLSR-Transducer ASR model for dynamic task activation. This core mechanism enables training with utterances labeled for only a subset of tasks, a key advantage over TokenVerse. We demonstrate this by successfully integrating a dataset with partial labels, specifically for ASR and an additional task, language identification, improving overall performance. TokenVerse++ achieves results on par with or exceeding TokenVerse across multiple tasks, establishing it as a more practical multitask alternative without sacrificing ASR performance.