# Input Combination Strategies for Multi-Source Transformer Decoder

arXiv: https://arxiv.org/abs/1811.04716 · HTML: https://mlsanity.com/paper/1811.04716/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2018.

## Built on by (most cited first, 22 of 22)

- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Alternative Input Signals Ease Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2110.07804/) (2021)
- [From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers](https://mlsanity.com/paper/2310.11984/) (2023)
- [HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representations](https://mlsanity.com/paper/2205.15479/) (2022)
- [AnyTrans: Translate AnyText in the Image with Large Scale Models](https://mlsanity.com/paper/2406.11432/) (2024)
- [Unsupervised domain adaptation for speech recognition with unsupervised error correction](https://mlsanity.com/paper/2209.12043/) (2022)
- [Visual Cues and Error Correction for Translation Robustness](https://mlsanity.com/paper/2103.07352/) (2021)
- [AlloST: Low-resource Speech Translation without Source Transcription](https://mlsanity.com/paper/2105.00171/) (2021)
- [Transfer Learning for Sequence Generation: from Single-source to Multi-source](https://mlsanity.com/paper/2105.14809/) (2021)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)
- [MTCue: Learning Zero-Shot Control of Extra-Textual Attributes by Leveraging Unstructured Context in Neural Machine Translation](https://mlsanity.com/paper/2305.15904/) (2023)
- [Enhancing conversational quality in language learning chatbots: An evaluation of GPT4 for ASR error correction](https://mlsanity.com/paper/2307.09744/) (2023)
- [Predicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream Transformer](https://mlsanity.com/paper/2103.08800/) (2021)
- [A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation](https://mlsanity.com/paper/2308.06063/) (2023)
- [Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation](https://mlsanity.com/paper/2312.15872/) (2023)
- [Multi-Source Transformer Architectures for Audiovisual Scene Classification](https://mlsanity.com/paper/2210.10212/) (2022)
- [Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis](https://mlsanity.com/paper/2409.08162/) (2024)
- [Contextual Cues in Machine Translation: Investigating the Potential of Multi-Source Input Strategies in LLMs and NMT Systems](https://mlsanity.com/paper/2503.07195/) (2025)

## Abstract (arXiv)

In multi-source sequence-to-sequence tasks, the attention mechanism can be modeled in several ways. This topic has been thoroughly studied on recurrent architectures. In this paper, we extend the previous work to the encoder-decoder attention in the Transformer architecture. We propose four different input combination strategies for the encoder-decoder attention: serial, parallel, flat, and hierarchical. We evaluate our methods on tasks of multimodal translation and translation with multiple source languages. The experiments show that the models are able to use multiple sources and improve over single source baselines.