# Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition

arXiv: https://arxiv.org/abs/2106.06922 · HTML: https://mlsanity.com/paper/2106.06922/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 13 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Achieving Human Parity in Conversational Speech Recognition](https://mlsanity.com/paper/1610.05256/) (2016)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification](https://mlsanity.com/paper/2004.05707/) (2020)
- [Innovative Bert-based Reranking Language Models for Speech Recognition](https://mlsanity.com/paper/2104.04950/) (2021)
- [Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion](https://mlsanity.com/paper/1906.11604/) (2019)
- [Cross-Attention End-to-End ASR for Two-Party Conversations](https://mlsanity.com/paper/1907.10726/) (2019)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)

## Built on by (most cited first, 7 of 7)

- [Automatic Speech Recognition with BERT and CTC Transformers: A Review](https://mlsanity.com/paper/2410.09456/) (2024)
- [Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR](https://mlsanity.com/paper/2207.01039/) (2022)
- [How Much Context Does My Attention-Based ASR System Need?](https://mlsanity.com/paper/2310.15672/) (2023)
- [Cross-utterance ASR Rescoring with Graph-based Label Propagation](https://mlsanity.com/paper/2303.15132/) (2023)
- [Leveraging Cross-Utterance Context For ASR Decoding](https://mlsanity.com/paper/2306.16903/) (2023)
- [Automated Speaking Assessment of Conversation Tests with Novel Graph-based Modeling on Spoken Response Coherence](https://mlsanity.com/paper/2409.07064/) (2024)
- [Effective Cross-Utterance Language Modeling for Conversational Speech Recognition](https://mlsanity.com/paper/2111.03333/) (2021)

## Abstract (arXiv)

How to effectively incorporate cross-utterance information cues into a neural language model (LM) has emerged as one of the intriguing issues for automatic speech recognition (ASR). Existing research efforts on improving contextualization of an LM typically regard previous utterances as a sequence of additional input and may fail to capture complex global structural dependencies among these utterances. In view of this, we in this paper seek to represent the historical context information of an utterance as graph-structured data so as to distill cross-utterances, global word interaction relationships. To this end, we apply a graph convolutional network (GCN) on the resulting graph to obtain the corresponding GCN embeddings of historical words. GCN has recently found its versatile applications on social-network analysis, text summarization, and among others due mainly to its ability of effectively capturing rich relational information among elements. However, GCN remains largely underexplored in the context of ASR, especially for dealing with conversational speech. In addition, we frame ASR N-best reranking as a prediction problem, leveraging bidirectional encoder representations from transformers (BERT) as the vehicle to not only seize the local intrinsic word regularity patterns inherent in a candidate hypothesis but also incorporate the cross-utterance, historical word interaction cues distilled by GCN for promoting performance. Extensive experiments conducted on the AMI benchmark dataset seem to confirm the pragmatic utility of our methods, in relation to some current top-of-the-line methods.