# Multiscale Collaborative Deep Models for Neural Machine Translation

arXiv: https://arxiv.org/abs/2004.14021 · HTML: https://mlsanity.com/paper/2004.14021/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation](https://mlsanity.com/paper/2104.02308/) (2021)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning](https://mlsanity.com/paper/2306.00103/) (2023)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation](https://mlsanity.com/paper/2212.12662/) (2022)
- [Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs](https://mlsanity.com/paper/2506.11515/) (2025)

## Abstract (arXiv)

Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neural networks can be more effective but are difficult to train. In this paper, we present a MultiScale Collaborative (MSC) framework to ease the training of NMT models that are substantially deeper than those used previously. We explicitly boost the gradient back-propagation from top to bottom levels by introducing a block-scale collaboration mechanism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration. We provide empirical evidence showing that the MSC nets are easy to optimize and can obtain improvements of translation quality from considerably increased depth. On IWSLT translation tasks with three translation directions, our extremely deep models (with 72-layer encoders) surpass strong baselines by +2.2~+3.1 BLEU points. In addition, our deep MSC achieves a BLEU score of 30.56 on WMT14 English-German task that significantly outperforms state-of-the-art deep NMT models.