# Multimodal Punctuation Prediction with Contextual Dropout

arXiv: https://arxiv.org/abs/2102.11012 · HTML: https://mlsanity.com/paper/2102.11012/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech](https://mlsanity.com/paper/2008.00702/) (2020)
- [Robust Prediction of Punctuation and Truecasing for Medical ASR](https://mlsanity.com/paper/2007.02025/) (2020)

## Built on by (most cited first, 1 of 1)

- [Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus](https://mlsanity.com/paper/2202.00468/) (2022)

## Abstract (arXiv)

Automatic speech recognition (ASR) is widely used in consumer electronics. ASR greatly improves the utility and accessibility of technology, but usually the output is only word sequences without punctuation. This can result in ambiguity in inferring user-intent. We first present a transformer-based approach for punctuation prediction that achieves 8% improvement on the IWSLT 2012 TED Task, beating the previous state of the art [1]. We next describe our multimodal model that learns from both text and audio, which achieves 8% improvement over the text-only algorithm on an internal dataset for which we have both the audio and transcriptions. Finally, we present an approach to learning a model using contextual dropout that allows us to handle variable amounts of future context at test time.