# RealFormer: Transformer Likes Residual Attention

arXiv: https://arxiv.org/abs/2012.11747 · HTML: https://mlsanity.com/paper/2012.11747/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 30 of 62)

- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Revisiting Over-smoothing in BERT from the Perspective of Graph](https://mlsanity.com/paper/2202.08625/) (2022)
- [Uncertainty-aware Vision-based Metric Cross-view Geolocalization](https://mlsanity.com/paper/2211.12145/) (2022)
- [ResiDual: Transformer with Dual Residual Connections](https://mlsanity.com/paper/2304.14802/) (2023)
- [EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm](https://mlsanity.com/paper/2206.09325/) (2022)
- [MedGPT: Medical Concept Prediction from Clinical Narratives](https://mlsanity.com/paper/2107.03134/) (2021)
- [OmniNet: Omnidirectional Representations from Transformers](https://mlsanity.com/paper/2103.01075/) (2021)
- [Graph Contrastive Learning for Anomaly Detection](https://mlsanity.com/paper/2108.07516/) (2021)
- [Cross-Layer Retrospective Retrieving via Layer Attention](https://mlsanity.com/paper/2302.03985/) (2023)
- [Layerwise Recurrent Router for Mixture-of-Experts](https://mlsanity.com/paper/2408.06793/) (2024)
- [Evolving Attention with Residual Convolutions](https://mlsanity.com/paper/2102.12895/) (2021)
- [Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model](https://mlsanity.com/paper/2105.15089/) (2021)
- [TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities](https://mlsanity.com/paper/2212.06385/) (2022)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [B2T Connection: Serving Stability and Performance in Deep Transformers](https://mlsanity.com/paper/2206.00330/) (2022)
- [SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understanding](https://mlsanity.com/paper/2504.03254/) (2025)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention](https://mlsanity.com/paper/2102.06431/) (2021)
- [Strengthening Layer Interaction via Dynamic Layer Attention](https://mlsanity.com/paper/2406.13392/) (2024)
- [An Effective Deployment of Contrastive Learning in Multi-label Text Classification](https://mlsanity.com/paper/2212.00552/) (2022)
- [Semantic Exploration with Adaptive Gating for Efficient Problem Solving with Language Models](https://mlsanity.com/paper/2501.05752/) (2025)
- [Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Method](https://mlsanity.com/paper/2601.20026/) (2026)
- [Delta Attention Residuals](https://mlsanity.com/paper/2605.18855/) (2026)
- [Searching a High-Performance Feature Extractor for Text Recognition Network](https://mlsanity.com/paper/2209.13139/) (2022)
- [MultiMax: Sparse and Multi-Modal Attention Learning](https://mlsanity.com/paper/2406.01189/) (2024)
- [Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention](https://mlsanity.com/paper/2103.15722/) (2021)

## Abstract (arXiv)

Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention. Source code and pre-trained checkpoints for RealFormer can be found at https://github.com/google-research/google-research/tree/master/realformer.