# Lattice Transformer for Speech Translation

arXiv: https://arxiv.org/abs/1906.05551 · HTML: https://mlsanity.com/paper/1906.05551/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2019.

## Built on by (most cited first, 15 of 15)

- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space](https://mlsanity.com/paper/2303.00456/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement](https://mlsanity.com/paper/2112.10991/) (2021)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models](https://mlsanity.com/paper/2407.15415/) (2024)
- [Adaptive Policy with Wait-k Model for Simultaneous Translation](https://mlsanity.com/paper/2310.14853/) (2023)
- [Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference](https://mlsanity.com/paper/2303.07914/) (2023)
- [An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2308.14482/) (2023)
- [Does Joint Training Really Help Cascaded Speech Translation?](https://mlsanity.com/paper/2210.13700/) (2022)
- [EEL: Efficiently Encoding Lattices for Reranking](https://mlsanity.com/paper/2306.00947/) (2023)
- [Improving speech translation by fusing speech and text](https://mlsanity.com/paper/2305.14042/) (2023)
- [The IWSLT 2021 BUT Speech Translation Systems](https://mlsanity.com/paper/2107.06155/) (2021)
- [Interpretable Style Takagi-Sugeno-Kang Fuzzy Clustering](https://mlsanity.com/paper/2504.05125/) (2025)

## Abstract (arXiv)

Recent advances in sequence modeling have highlighted the strengths of the transformer architecture, especially in achieving state-of-the-art machine translation results. However, depending on the up-stream systems, e.g., speech recognition, or word segmentation, the input to translation system can vary greatly. The goal of this work is to extend the attention mechanism of the transformer to naturally consume the lattice in addition to the traditional sequential input. We first propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) which contains multiple paths and posterior scores. To leverage the extra information from the lattice structure, we develop a novel controllable lattice attention mechanism to obtain latent representations. On the LDC Spanish-English speech translation corpus, our experiments show that lattice transformer generalizes significantly better and outperforms both a transformer baseline and a lattice LSTM. Additionally, we validate our approach on the WMT 2017 Chinese-English translation task with lattice inputs from different BPE segmentations. In this task, we also observe the improvements over strong baselines.