mlsanityData to 14 Oct 2026

Graph Convolutions Enrich the Self-Attention in Transformers!

arXiv 2312.04234 ↗First posted 7 Dec 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose a graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph-level tasks, speech recognition, and code classification.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 83 of its 109 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

83 of 109 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  11. 2017Graph Attention Networkscited by 8,923
  12. 2016Densely Connected Convolutional Networkscited by 6,720
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  7. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  13. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  15. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  16. 2018Neural Network Acceptability Judgmentscited by 958
  17. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  18. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  19. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
  20. 2021Going deeper with Image Transformerscited by 764
  21. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  22. 2020Benchmarking Graph Neural Networkscited by 680
  23. 2019Green AIcited by 627
  24. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  25. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  26. 2019Diffusion Improves Graph Learningcited by 414
  27. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  28. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  29. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  30. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  31. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  32. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  33. 2022Long Range Graph Benchmarkcited by 263
  34. 2021Global Filter Networks for Image Classificationcited by 260
  35. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 249
  36. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
  37. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 231
  38. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 4; 2026: 4.

11 papers.

  1. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  2. 2025Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domaincited by 4
  3. 2023RDGCL: Reaction-Diffusion Graph Contrastive Learning for Recommendationcited by 3
  4. 2025Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvementscited by 3
  5. 2025Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNscited by 3
  6. 2026Capacity-Controlled Global Attention for Graph Transformerscited by 2
  7. 2024Graph Contrastive Learning with Low-Rank Regularization and Low-Rank Attention for Noisy Node Classificationcited by 1
  8. 2025TV-Rec: Time-Variant Convolutional Filter for Sequential Recommendationcited by 0
  9. 2026Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulationcited by 0
  10. 2026FAiT: Frequency-Aware Inverted Transformer for Multivariate Time Series Forecastingcited by 0
  11. 2026Difference-of-Convex Regularization for Graph Learning by Differentiable Programmingcited by 0