# Rethinking the adaptive relationship between Encoder Layers and Decoder Layers

arXiv: https://arxiv.org/abs/2405.08570 · HTML: https://mlsanity.com/paper/2405.08570/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 2 references that are in the map.
- First posted on arXiv 14 May 2024.

## Builds on (most cited first, 2 of 2)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)

## Built on by (most cited first, 1 of 1)

- [Thread-Efficient Decoding for Neural Texture Compression](https://mlsanity.com/paper/2608.27888/) (2026)

## Abstract (arXiv)

This article explores the adaptive relationship between Encoder Layers and Decoder Layers using the SOTA model Helsinki-NLP/opus-mt-de-en, which translates German to English. The specific method involves introducing a bias-free fully connected layer between the Encoder and Decoder, with different initializations of the layer's weights, and observing the outcomes of fine-tuning versus retraining. Four experiments were conducted in total. The results suggest that directly modifying the pre-trained model structure for fine-tuning yields suboptimal performance. However, upon observing the outcomes of the experiments with retraining, this structural adjustment shows significant potential.