mlsanityData to 7 Oct 2026

RealFormer: Transformer Likes Residual Attention

arXiv 2012.11747 ↗First posted 21 Dec 2020cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention. Source code and pre-trained checkpoints for RealFormer can be found at https://github.com/google-research/google-research/tree/master/realformer.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 14; 2024: 5; 2025: 7; 2026: 8.

62 papers.

Sort:

By area:

  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2021A Survey of Transformerscited by 399
  4. 2021Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networkscited by 176
  5. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  6. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  7. 2022Uncertainty-aware Vision-based Metric Cross-view Geolocalizationcited by 54
  8. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  9. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  10. 2021MedGPT: Medical Concept Prediction from Clinical Narrativescited by 24
  11. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  12. 2021Graph Contrastive Learning for Anomaly Detectioncited by 22
38 more
  1. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  2. 2021Evolving Attention with Residual Convolutionscited by 20
  3. 2021Analogous to Evolutionary Algorithm: Designing a Unified Sequence Modelcited by 20
  4. 2023Cross-Layer Retrospective Retrieving via Layer Attentioncited by 20
  5. 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
  6. 2021Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Modelcited by 18
  7. 2022B2T Connection: Serving Stability and Performance in Deep Transformerscited by 16
  8. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understandingcited by 16
  9. 2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
  10. 2021VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attentioncited by 12
  11. 2024Strengthening Layer Interaction via Dynamic Layer Attentioncited by 10
  12. 2022An Effective Deployment of Contrastive Learning in Multi-label Text Classificationcited by 9
  13. 2025Semantic Exploration with Adaptive Gating for Efficient Problem Solving with Language Modelscited by 7
  14. 2026Delta Attention Residualscited by 7
  15. 2024MultiMax: Sparse and Multi-Modal Attention Learningcited by 5
  16. 2021Transformer-based end-to-end speech recognition with residual Gaussian-based self-attentioncited by 4
  17. 2022Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanismcited by 4
  18. 2022Searching a High-Performance Feature Extractor for Text Recognition Networkcited by 4
  19. 2023HAGNN: Hybrid Aggregation for Heterogeneous Graph Neural Networkscited by 4
  20. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4
  21. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  22. 2026Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Methodcited by 4
  23. 2022N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Modelcited by 3
  24. 2023LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehensioncited by 3
  25. 2026Depth-Attention: Cross-Layer Value Mixing for Language Modelscited by 3
  26. 2021TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidancecited by 2
  27. 2022Convolution-enhanced Evolving Attention Networkscited by 2
  28. 2023Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognitioncited by 2
  29. 2026Multi-Head Attention Residualscited by 2
  30. 2021MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similaritycited by 1
  31. 2022Relational Graph Convolutional Neural Networks for Multihop Reasoning: A Comparative Studycited by 1
  32. 2023ResMatch: Residual Attention Learning for Local Feature Matchingcited by 1
  33. 2023Integrating a Heterogeneous Graph with Entity-aware Self-attention using Relative Position Labels for Reading Comprehension Modelcited by 1
  34. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  35. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  36. 2025Advances and Challenges in Semantic Textual Similarity: A Comprehensive Surveycited by 1
  37. 2021Cascaded Head-colliding Attentioncited by 0
  38. 2022Identifying the root cause of cable network problems with machine learningcited by 0

50 most cited of 62 All 62 (JSON)