# Lattention: Lattice-attention in ASR rescoring

arXiv: https://arxiv.org/abs/2111.10157 · HTML: https://mlsanity.com/paper/2111.10157/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Deliberation Model Based Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/2003.07962/) (2020)
- [Neural Lattice-to-Sequence Models for Uncertain Inputs](https://mlsanity.com/paper/1704.00559/) (2017)
- [Audio-attention discriminative language model for ASR rescoring](https://mlsanity.com/paper/1912.03363/) (2019)
- [Self-Attentional Models for Lattice Inputs](https://mlsanity.com/paper/1906.01617/) (2019)
- [Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation](https://mlsanity.com/paper/1609.07730/) (2016)
- [Lattice-Based Transformer Encoder for Neural Machine Translation](https://mlsanity.com/paper/1906.01282/) (2019)

## Built on by (most cited first, 6 of 6)

- [EEL: Efficiently Encoding Lattices for Reranking](https://mlsanity.com/paper/2306.00947/) (2023)
- [Transformer-based Model for ASR N-Best Rescoring and Rewriting](https://mlsanity.com/paper/2406.08207/) (2024)
- [Personalization for BERT-based Discriminative Speech Recognition Rescoring](https://mlsanity.com/paper/2307.06832/) (2023)
- [Leveraging Cross-Utterance Context For ASR Decoding](https://mlsanity.com/paper/2306.16903/) (2023)
- [Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring](https://mlsanity.com/paper/2310.09680/) (2023)
- [Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing](https://mlsanity.com/paper/2509.16622/) (2025)

## Abstract (arXiv)

Lattices form a compact representation of multiple hypotheses generated from an automatic speech recognition system and have been shown to improve performance of downstream tasks like spoken language understanding and speech translation, compared to using one-best hypothesis. In this work, we look into the effectiveness of lattice cues for rescoring n-best lists in second-pass. We encode lattices with a recurrent network and train an attention encoder-decoder model for n-best rescoring. The rescoring model with attention to lattices achieves 4-5% relative word error rate reduction over first-pass and 6-8% with attention to both lattices and acoustic features. We show that rescoring models with attention to lattices outperform models with attention to n-best hypotheses. We also study different ways to incorporate lattice weights in the lattice encoder and demonstrate their importance for n-best rescoring.