# Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network

arXiv: https://arxiv.org/abs/2302.13376 · HTML: https://mlsanity.com/paper/2302.13376/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech](https://mlsanity.com/paper/2008.00702/) (2020)
- [Token-Level Supervised Contrastive Learning for Punctuation Restoration](https://mlsanity.com/paper/2107.09099/) (2021)
- [Improving Vietnamese Named Entity Recognition from Speech Using Word Capitalization and Punctuation Recovery Models](https://mlsanity.com/paper/2010.00198/) (2020)
- [Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus](https://mlsanity.com/paper/2202.00468/) (2022)
- [Vietnamese Capitalization and Punctuation Recovery Models](https://mlsanity.com/paper/2207.01312/) (2022)

## Built on by (most cited first, 1 of 1)

- [Spontaneous Informal Speech Dataset for Punctuation Restoration](https://mlsanity.com/paper/2409.11241/) (2024)

## Abstract (arXiv)

Punctuation restoration plays an essential role in the post-processing procedure of automatic speech recognition, but model efficiency is a key requirement for this task. To that end, we present EfficientPunct, an ensemble method with a multimodal time-delay neural network that outperforms the current best model by 1.0 F1 points, using less than a tenth of its inference network parameters. We streamline a speech recognizer to efficiently output hidden layer acoustic embeddings for punctuation restoration, as well as BERT to extract meaningful text embeddings. By using forced alignment and temporal convolutions, we eliminate the need for attention-based fusion, greatly increasing computational efficiency and raising performance. EfficientPunct sets a new state of the art with an ensemble that weights BERT's purely language-based predictions slightly more than the multimodal network's predictions. Our code is available at https://github.com/lxy-peter/EfficientPunct.