mlsanityData to 7 Oct 2026

Designing Robust Transformers using Robust Kernel Density Estimation

arXiv 2210.05794 ↗First posted 11 Oct 2022cs.LG cs.CL cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advances in Transformer architectures have empowered their empirical success in a variety of tasks across different domains. However, existing works mainly focus on predictive accuracy and computational cost, without considering other practical issues, such as robustness to contaminated samples. Recent work by Nguyen et al., (2022) has shown that the self-attention mechanism, which is the center of the Transformer architecture, can be viewed as a non-parametric estimator based on kernel density estimation (KDE). This motivates us to leverage a set of robust kernel density estimation methods for alleviating the issue of data contamination. Specifically, we introduce a series of self-attention mechanisms that can be incorporated into different Transformer architectures and discuss the special properties of each method. We then perform extensive empirical studies on language modeling and image classification tasks. Our methods demonstrate robust performance in multiple scenarios while maintaining competitive results on clean datasets.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 59 of its 85 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

59 of 85 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  11. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2018Self-Attention with Relative Position Representationscited by 1,112
  15. 2021Video Swin Transformercited by 1,090
  16. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  17. 2021Transformers in Vision: A Surveycited by 1,009
  18. 2021Multiscale Vision Transformerscited by 896
  19. 2020Efficient Transformers: A Surveycited by 784
  20. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  21. 2018Universal Transformerscited by 613
  22. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  23. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  24. 2021A Survey of Transformerscited by 399
  25. 2021Choose a Transformer: Fourier or Galerkincited by 333
  26. 2018The UEA multivariate time series classification archive, 2018cited by 330
  27. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  28. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  29. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  30. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  31. 2021Vision Transformers are Robust Learnerscited by 208
  32. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  33. 2022Understanding The Robustness in Vision Transformerscited by 144
  34. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  35. 2021Towards Robust Vision Transformercited by 137
  36. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  37. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  38. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 90

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 2.

11 papers.

  1. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  2. 2025Tight Clusters Make Specialized Expertscited by 8
  3. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  4. 2024Elliptical Attentioncited by 7
  5. 2024ProTransformer: Robustify Transformers via Plug-and-Play Paradigmcited by 5
  6. 2024AdaNCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformercited by 4
  7. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  8. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  9. 2025Robust Filter Attention: Self-Attention as Precision-Weighted State Estimationcited by 1
  10. 2026QUEST: A robust attention formulation using query-modulated spherical attentioncited by 1
  11. 2026Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attentioncited by 0