# A Fully Differentiable Beam Search Decoder

arXiv: https://arxiv.org/abs/1902.06022 · HTML: https://mlsanity.com/paper/1902.06022/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2019.

## Built on by (most cited first, 14 of 14)

- [Machine Translation Decoding beyond Beam Search](https://mlsanity.com/paper/2104.05336/) (2021)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [Learning with Algorithmic Supervision via Continuous Relaxations](https://mlsanity.com/paper/2110.05651/) (2021)
- [Best Practices for a Handwritten Text Recognition System](https://mlsanity.com/paper/2404.11339/) (2024)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Incremental Beam Manipulation for Natural Language Generation](https://mlsanity.com/paper/2102.02574/) (2021)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Silent Speech and Emotion Recognition from Vocal Tract Shape Dynamics in Real-Time MRI](https://mlsanity.com/paper/2106.08706/) (2021)
- [Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach](https://mlsanity.com/paper/2110.10429/) (2021)
- [Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction](https://mlsanity.com/paper/2308.08442/) (2023)
- [Agentic AI: The Era of Semantic Decoding](https://mlsanity.com/paper/2403.14562/) (2024)
- [Understanding and Enhancing the Use of Context for Machine Translation](https://mlsanity.com/paper/2102.10437/) (2021)
- [O-1: Self-training with Oracle and 1-best Hypothesis](https://mlsanity.com/paper/2308.07486/) (2023)

## Abstract (arXiv)

We introduce a new beam search decoder that is fully differentiable, making it possible to optimize at training time through the inference procedure. Our decoder allows us to combine models which operate at different granularities (e.g. acoustic and language models). It can be used when target sequences are not aligned to input sequences by considering all possible alignments between the two. We demonstrate our approach scales by applying it to speech recognition, jointly training acoustic and word-level language models. The system is end-to-end, with gradients flowing through the whole architecture from the word-level transcriptions. Recent research efforts have shown that deep neural networks with attention-based mechanisms are powerful enough to successfully train an acoustic model from the final transcription, while implicitly learning a language model. Instead, we show that it is possible to discriminatively train an acoustic model jointly with an explicit and possibly pre-trained language model.