mlsanityData to 14 Oct 2026

Unveiling Invariances via Neural Network Pruning

arXiv 2309.08171 ↗First posted 15 Sep 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Invariance describes transformations that do not alter data's underlying semantics. Neural networks that preserve natural invariance capture good inductive biases and achieve superior performance. Hence, modern networks are handcrafted to handle well-known invariances (ex. translations). We propose a framework to learn novel network architectures that capture data-dependent invariances via pruning. Our learned architectures consistently outperform dense neural networks on both vision and tabular datasets in both efficiency and effectiveness. We demonstrate our framework on multiple deep learning models across 3 vision and 40 tabular datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 70 of its 89 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

70 of 89 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  12. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
38 more
  1. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  4. 2015Spatial Transformer Networkscited by 1,429
  5. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  6. 2016Group Equivariant Convolutional Networkscited by 1,165
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  8. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  9. 2021E(n) Equivariant Graph Neural Networkscited by 965
  10. 2017A Closer Look at Memorization in Deep Networkscited by 942
  11. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  12. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  13. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  14. 2017Dynamic Routing Between Capsulescited by 689
  15. 2018On Lazy Training in Differentiable Programmingcited by 675
  16. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  17. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  18. 2018Rethinking the Value of Network Pruningcited by 599
  19. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  20. 2020What is the State of Neural Network Pruning?cited by 583
  21. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  22. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  23. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  24. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  25. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  26. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  27. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  28. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  29. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  30. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  31. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  32. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  33. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  34. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  35. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  36. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  37. 2018Neural Architecture Optimizationcited by 167
  38. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.