mlsanityData to 14 Oct 2026

Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets

arXiv 2206.11925 ↗First posted 23 Jun 2022cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Permutation invariant neural networks are a promising tool for making predictions from sets. However, we show that existing permutation invariant architectures, Deep Sets and Set Transformer, can suffer from vanishing or exploding gradients when they are deep. Additionally, layer norm, the normalization of choice in Set Transformer, can hurt performance by removing information useful for prediction. To address these issues, we introduce the clean path principle for equivariant residual connections and develop set norm, a normalization tailored for sets. With these, we build Deep Sets++ and Set Transformer++, models that reach high depths with comparable or better performance than their original counterparts on a diverse suite of tasks. We additionally introduce Flow-RBC, a new single-cell dataset and real-world application of permutation invariant prediction. We open-source our data and code here: https://github.com/rajesh-lab/deep_permutation_invariant.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  9. 2017Deep Setscited by 1,421
  10. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  11. 2020On Layer Normalization in the Transformer Architecturecited by 922
  12. 2020Simple and Deep Graph Convolutional Networkscited by 860
18 more
  1. 2019Deep Learning for 3D Point Clouds: A Surveycited by 558
  2. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  3. 2019General E(2)-Equivariant Steerable CNNscited by 446
  4. 2018How Does Batch Normalization Help Optimization?cited by 357
  5. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  6. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  8. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  9. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  10. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  11. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  12. 2020Incorporating Symmetry into Deep Dynamics Models for Improved Generalizationcited by 126
  13. 2020GraphNorm: A Principled Approach to Accelerating Graph Neural Network Trainingcited by 117
  14. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 102
  15. 2019On the Limitations of Representing Functions on Setscited by 90
  16. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  17. 2019A Mean Field Theory of Batch Normalizationcited by 58
  18. 2018Towards Understanding Regularization in Batch Normalizationcited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 3.

17 papers.

Sort:

By area:

  1. 2023MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generationcited by 98
  2. 2024Wasserstein Flow Matching: Generative modeling over families of distributionscited by 28
  3. 2025Black Box Causal Inference: Effect Estimation via Meta Predictioncited by 13
  4. 2024On permutation-invariant neural networkscited by 10
  5. 2025Generative Distribution Embeddings: Lifting autoencoders to the space of distributions for multiscale representation learningcited by 4
  6. 2025Advances in Set Function Learning: A Survey of Techniques and Applicationscited by 3
  7. 2025Meta-Statistical Learning: Supervised Learning of Statistical Estimatorscited by 3
  8. 2024UnitNorm: Rethinking Normalization for Transformers in Time Seriescited by 2
  9. 2025Scalable Single-Cell Gene Expression Generation with Latent Diffusion Modelscited by 2
  10. 2026Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regressioncited by 2
  11. 2026Distribution-Conditioned Transportcited by 2
  12. 2024Learning Set Functions with Implicit Differentiationcited by 1
5 more
  1. 2025MIST: Mutual Information Estimation Via Supervised Trainingcited by 1
  2. 2024Set2Seq Transformer: Temporal and Position-Aware Set Representations for Sequential Multiple-Instance Learningcited by 0
  3. 2025Enhancing Job Salary Prediction with Disentangled Composition Effect Modeling: A Neural Prototyping Approachcited by 0
  4. 2025CIGaRS I: Combined simulation-based inference from type Ia supernovae and host photometrycited by 0
  5. 2026Improving Set Function Approximation with Quasi-Arithmetic Neural Networkscited by 0