# Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

arXiv: https://arxiv.org/abs/2004.08178 · HTML: https://mlsanity.com/paper/2004.08178/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2020.

## Built on by (most cited first, 18 of 18)

- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages](https://mlsanity.com/paper/2212.06742/) (2022)
- [Residual Matrix Transformers: Scaling the Size of the Residual Stream](https://mlsanity.com/paper/2506.22696/) (2025)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [Autoregressive Pre-Training on Pixels and Texts](https://mlsanity.com/paper/2404.10710/) (2024)
- [Deep Delta Learning](https://mlsanity.com/paper/2601.00417/) (2026)
- [Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks](https://mlsanity.com/paper/2210.00185/) (2022)
- [InterFormer: Effective Heterogeneous Interaction Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2411.09852/) (2024)
- [Investigating Recurrent Transformers with Dynamic Halt](https://mlsanity.com/paper/2402.00976/) (2024)
- [On the Design Space Between Transformers and Recursive Neural Nets](https://mlsanity.com/paper/2409.01531/) (2024)
- [GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization](https://mlsanity.com/paper/2601.22095/) (2026)
- [Leveraging Local Temporal Information for Multimodal Scene Classification](https://mlsanity.com/paper/2110.13992/) (2021)
- [Can Transformers Predict Vibrations?](https://mlsanity.com/paper/2402.10511/) (2024)
- [From Features to Transformers: Redefining Ranking for Scalable Impact](https://mlsanity.com/paper/2502.03417/) (2025)
- [IgCraft: A versatile sequence generation framework for antibody discovery and engineering](https://mlsanity.com/paper/2503.19821/) (2025)
- [EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs](https://mlsanity.com/paper/2509.16686/) (2025)
- [CADET: Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformer](https://mlsanity.com/paper/2602.11410/) (2026)
- [GateMOT: Q-Gated Attention for Dense Object Tracking](https://mlsanity.com/paper/2604.26353/) (2026)

## Abstract (arXiv)

Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. The subsidiary content-based SDU gates allow for the information flow of modulated latent embeddings through skipped connections, leading to a clear margin of convergence speed with gradient descent algorithms. We may unveil the role of gating mechanism to aid in the context-based Transformer modules, with hypothesizing that SDU gates, especially on shallow layers, could push it faster to step towards suboptimal points during the optimization process.