# Factual Consistency Oriented Speech Recognition

arXiv: https://arxiv.org/abs/2302.12369 · HTML: https://mlsanity.com/paper/2302.12369/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Achieving Human Parity in Conversational Speech Recognition](https://mlsanity.com/paper/1610.05256/) (2016)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021)
- [Lessons from Building Acoustic Models with a Million Hours of Speech](https://mlsanity.com/paper/1904.01624/) (2019)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [Speech Summarization using Restricted Self-Attention](https://mlsanity.com/paper/2110.06263/) (2021)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding](https://mlsanity.com/paper/2102.06750/) (2021)

## Built on by (most cited first, 1 of 1)

- [MGSC: A Multi-granularity Consistency Framework for Robust End-to-end Asr](https://mlsanity.com/paper/2508.15853/) (2025)

## Abstract (arXiv)

This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.