# Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

arXiv: https://arxiv.org/abs/2012.14768 · HTML: https://mlsanity.com/paper/2012.14768/

## Key facts

- Cited by 26 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 26 of 26)

- [Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models](https://mlsanity.com/paper/2210.05497/) (2022)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers](https://mlsanity.com/paper/2310.03686/) (2023)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [Revisiting Grammatical Error Correction Evaluation and Beyond](https://mlsanity.com/paper/2211.01635/) (2022)
- [Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2201.04831/) (2022)
- [On the Copying Behaviors of Pre-Training for Neural Machine Translation](https://mlsanity.com/paper/2107.08212/) (2021)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Self-Guided Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2105.04475/) (2021)
- [Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification](https://mlsanity.com/paper/2209.15168/) (2022)
- [VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing](https://mlsanity.com/paper/2605.30117/) (2026)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modeling](https://mlsanity.com/paper/2211.12781/) (2022)
- [Improving Neural Machine Translation by Denoising Training](https://mlsanity.com/paper/2201.07365/) (2022)
- [Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention](https://mlsanity.com/paper/2203.06822/) (2022)
- [Prompt-Learning for Cross-Lingual Relation Extraction](https://mlsanity.com/paper/2304.10354/) (2023)
- [Self-Powered LLM Modality Expansion for Large Speech-Text Models](https://mlsanity.com/paper/2410.03798/) (2024)
- [Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning](https://mlsanity.com/paper/2411.03042/) (2024)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [An approach to optimize inference of the DIART speaker diarization pipeline](https://mlsanity.com/paper/2408.02341/) (2024)
- [Lightweight Adaptive Feature Composition for Heterogeneous Downstream Adaptation of Wireless Foundation Models](https://mlsanity.com/paper/2606.10277/) (2026)
- [Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages](https://mlsanity.com/paper/2508.09091/) (2025)
- [MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling](https://mlsanity.com/paper/2603.03001/) (2026)
- [WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs](https://mlsanity.com/paper/2608.22704/) (2026)

## Abstract (arXiv)

Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embedding. Source code is freely available at https://github.com/SunbowLiu/SurfaceFusion.