# On Sparsifying Encoder Outputs in Sequence-to-Sequence Models

arXiv: https://arxiv.org/abs/2004.11854 · HTML: https://mlsanity.com/paper/2004.11854/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 8 of 8)

- [Learning When to Translate for Streaming Speech](https://mlsanity.com/paper/2109.07368/) (2021)
- [MultiMax: Sparse and Multi-Modal Attention Learning](https://mlsanity.com/paper/2406.01189/) (2024)
- [Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model](https://mlsanity.com/paper/2510.26622/) (2025)
- [Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention](https://mlsanity.com/paper/2211.07164/) (2022)
- [Learning to Group Auxiliary Datasets for Molecule](https://mlsanity.com/paper/2307.04052/) (2023)
- [Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention](https://mlsanity.com/paper/2102.01223/) (2021)
- [Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation](https://mlsanity.com/paper/2110.07002/) (2021)
- [On the Relationship between Skill Neurons and Robustness in Prompt Tuning](https://mlsanity.com/paper/2309.12263/) (2023)

## Abstract (arXiv)

Sequence-to-sequence models usually transfer all encoder outputs to the decoder for generation. In this work, by contrast, we hypothesize that these encoder outputs can be compressed to shorten the sequence delivered for decoding. We take Transformer as the testbed and introduce a layer of stochastic gates in-between the encoder and the decoder. The gates are regularized using the expected value of the sparsity-inducing L0penalty, resulting in completely masking-out a subset of encoder outputs. In other words, via joint training, the L0DROP layer forces Transformer to route information through a subset of its encoder states. We investigate the effects of this sparsification on two machine translation and two summarization tasks. Experiments show that, depending on the task, around 40-70% of source encodings can be pruned without significantly compromising quality. The decrease of the output length endows L0DROP with the potential of improving decoding efficiency, where it yields a speedup of up to 1.65x on document summarization tasks against the standard Transformer. We analyze the L0DROP behaviour and observe that it exhibits systematic preferences for pruning certain word types, e.g., function words and punctuation get pruned most. Inspired by these observations, we explore the feasibility of specifying rule-based patterns that mask out encoder outputs based on information such as part-of-speech tags, word frequency and word position.