mlsanityData to 7 Oct 2026

Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

arXiv 2106.02584 ↗First posted 4 Jun 2021cs.LG stat.MLcited by 112

Not on the map. Why?

Abstract (arXiv)

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 56 of its 114 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

56 of 114 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2017Graph Attention Networkscited by 8,854
  8. 2016Layer Normalizationcited by 5,579
  9. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  10. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2018How Powerful are Graph Neural Networks?cited by 4,355
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2016Matching Networks for One Shot Learningcited by 2,246
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2020Object-Centric Learning with Slot Attentioncited by 865
  12. 2020Efficient Transformers: A Surveycited by 784
  13. 2018Image Transformercited by 770
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  16. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  17. 2021ImageNet-21K Pretraining for the Massescited by 607
  18. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  19. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  20. 2015Deep Kernel Learningcited by 440
  21. 2018Conditional Neural Processescited by 421
  22. 2019Axial Attention in Multidimensional Transformerscited by 350
  23. 2018Neural Relational Inference for Interacting Systemscited by 339
  24. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  25. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  26. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  27. 2019Attentive Neural Processescited by 280
  28. 2017No Fuss Distance Metric Learning using Proxiescited by 225
  29. 2019Fixing the train-test resolution discrepancycited by 209
  30. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  31. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  32. 2020A Unified View of Label Shift Estimationcited by 119
  33. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  34. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  35. 2017Generating Sentences by Editing Prototypescited by 88
  36. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  37. 2019Ranked List Loss for Deep Metric Learningcited by 77
  38. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 24; 2023: 28; 2024: 26; 2025: 24; 2026: 8.

112 papers.

Sort:

By area:

  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  4. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  5. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  6. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  7. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  8. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  9. 2022Relative representations enable zero-shot latent space communicationcited by 175
  10. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  11. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  12. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
38 more
  1. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  2. 2022Amortized Inference for Causal Structure Learningcited by 89
  3. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  4. 2022Learning to Induce Causal Structurecited by 68
  5. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  6. 2022Transfer Learning with Deep Tabular Modelscited by 52
  7. 2022Neural Diffusion Processescited by 50
  8. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  9. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  10. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 37
  11. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  12. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  13. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  14. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 23
  15. 2022Revisiting a kNN-based Image Classification System with High-capacity Storagecited by 22
  16. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  17. 2022Multivariate Time Series Forecasting with Latent Graph Inferencecited by 20
  18. 2023Context-Aware Meta-Learningcited by 20
  19. 2024Contextual Document Embeddingscited by 19
  20. 2022Hopular: Modern Hopfield Networks for Tabular Datacited by 18
  21. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  22. 2024A Survey on Deep Tabular Learningcited by 18
  23. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  24. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  25. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  26. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  27. 2023Beyond Individual Input for Deep Anomaly Detection on Tabular Datacited by 14
  28. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  29. 2023Deep Clustering of Tabular Data by Weighted Gaussian Distribution Learningcited by 12
  30. 2023One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Datacited by 12
  31. 2022Deep Causal Learning: Representation, Discovery and Inferencecited by 11
  32. 20244DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBscited by 11
  33. 2025Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learningcited by 11
  34. 2023GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Datacited by 10
  35. 2024Amortized Active Causal Induction with Deep Reinforcement Learningcited by 10
  36. 2024Demystifying amortized causal discovery with transformerscited by 10
  37. 2024Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarkingcited by 9
  38. 2024Evolution Transformer: In-Context Evolutionary Optimizationcited by 9

50 most cited of 112 All 112 (JSON)