# UFO2: A unified pre-training framework for online and offline speech recognition

arXiv: https://arxiv.org/abs/2210.14515 · HTML: https://mlsanity.com/paper/2210.14515/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2022.

## Built on by (most cited first, 7 of 7)

- [ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents](https://mlsanity.com/paper/2604.11784/) (2026)
- [Decoupled Structure for Improved Adaptability of End-to-End Models](https://mlsanity.com/paper/2308.13345/) (2023)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training](https://mlsanity.com/paper/2409.08680/) (2024)
- [Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training](https://mlsanity.com/paper/2312.01515/) (2023)
- [Sparse Generation: Making Pseudo Labels Sparse for Point Weakly Supervised Object Detection on Low Data Volume](https://mlsanity.com/paper/2403.19306/) (2024)
- [HydraFormer: One Encoder For All Subsampling Rates](https://mlsanity.com/paper/2408.04325/) (2024)

## Abstract (arXiv)

In this paper, we propose a Unified pre-training Framework for Online and Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two separate training workflows for online and offline modes into one process, and 2) improves the Word Error Rate (WER) performance with limited utterance annotating. Specifically, we extend the conventional offline-mode Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where the model training is conditioned on both the full-context and dynamic-chunked inputs. To enhance the pre-trained representation model, stop-gradient operation is applied to decouple the online-mode objectives to the quantizer. Moreover, in both the pre-training and the downstream fine-tuning stages, joint losses are proposed to train the unified model with full-weight sharing for the two modes. Experimental results on the LibriSpeech dataset show that UFO2 outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER reduction in offline and online modes, respectively.