# Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech

arXiv: https://arxiv.org/abs/2008.00702 · HTML: https://mlsanity.com/paper/2008.00702/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models](https://mlsanity.com/paper/2310.02943/) (2023)
- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [End-to-end Speech-to-Punctuated-Text Recognition](https://mlsanity.com/paper/2207.03169/) (2022)
- [ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling](https://mlsanity.com/paper/2106.09532/) (2021)
- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)
- [Incorporating External POS Tagger for Punctuation Restoration](https://mlsanity.com/paper/2106.06731/) (2021)
- [Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus](https://mlsanity.com/paper/2202.00468/) (2022)
- [Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15063/) (2022)
- [Evaluating OpenAI's Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person](https://mlsanity.com/paper/2305.14580/) (2023)
- [Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation](https://mlsanity.com/paper/2306.01296/) (2023)
- [SemiMemes: A Semi-supervised Learning Approach for Multimodal Memes Analysis](https://mlsanity.com/paper/2304.00020/) (2023)
- [Resolving Transcription Ambiguity in Spanish: A Hybrid Acoustic-Lexical System for Punctuation Restoration](https://mlsanity.com/paper/2402.03519/) (2024)
- [Multimodal Punctuation Prediction with Contextual Dropout](https://mlsanity.com/paper/2102.11012/) (2021)
- [Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network](https://mlsanity.com/paper/2302.13376/) (2023)
- [A Small and Fast BERT for Chinese Medical Punctuation Restoration](https://mlsanity.com/paper/2308.12568/) (2023)
- [Spontaneous Informal Speech Dataset for Punctuation Restoration](https://mlsanity.com/paper/2409.11241/) (2024)
- [Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization](https://mlsanity.com/paper/2512.18967/) (2025)

## Abstract (arXiv)

In this work, we explore a multimodal semi-supervised learning approach for punctuation prediction by learning representations from large amounts of unlabelled audio and text data. Conventional approaches in speech processing typically use forced alignment to encoder per frame acoustic features to word level features and perform multimodal fusion of the resulting acoustic and lexical representations. As an alternative, we explore attention based multimodal fusion and compare its performance with forced alignment based fusion. Experiments conducted on the Fisher corpus show that our proposed approach achieves ~6-9% and ~3-4% absolute improvement (F1 score) over the baseline BLSTM model on reference transcripts and ASR outputs respectively. We further improve the model robustness to ASR errors by performing data augmentation with N-best lists which achieves up to an additional ~2-6% improvement on ASR outputs. We also demonstrate the effectiveness of semi-supervised learning approach by performing ablation study on various sizes of the corpus. When trained on 1 hour of speech and text data, the proposed model achieved ~9-18% absolute improvement over baseline model.