mlsanityData to 7 Oct 2026

Towards Learning Convolutions from Scratch

arXiv 2007.13657 ↗First posted 27 Jul 2020cs.LG cs.CV stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

Convolution is one of the most essential components of architectures used in computer vision. As machine learning moves towards reducing the expert bias and learning it from data, a natural next step seems to be learning convolution-like structures from scratch. This, however, has proven elusive. For example, current state-of-the-art architecture search algorithms use convolution as one of the existing modules rather than learning it from data. In an attempt to understand the inductive bias that gives rise to convolutions, we investigate minimum description length as a guiding principle and show that in some settings, it can indeed be indicative of the performance of architectures. To find architectures with small description length, we propose β-LASSO, a simple variant of LASSO algorithm that, when applied on fully-connected networks for image classification tasks, learns architectures with local connections and achieves state-of-the-art accuracies for training fully-connected nets on CIFAR-10 (85.19%), CIFAR-100 (59.56%) and SVHN (94.07%) bridging the gap between fully-connected and convolutional nets.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 13; 2024: 3; 2025: 1; 2026: 1.

46 papers.

Sort:

  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  3. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  4. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  5. 2021Vision Transformer for Small-Size Datasetscited by 122
  6. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  7. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  8. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  9. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  10. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  11. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  12. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
34 more
  1. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  2. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  3. 2022AA-TransUNet: Attention Augmented TransUNet For Nowcasting Taskscited by 22
  4. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  5. 2021Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networkscited by 14
  6. 2021Towards Biologically Plausible Convolutional Networkscited by 13
  7. 2023A Reliable and Interpretable Framework of Multi-view Learning for Liver Fibrosis Stagingcited by 13
  8. 2024MERIT: Multi-view evidential learning for reliable and interpretable liver fibrosis stagingcited by 12
  9. 2023ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imagingcited by 9
  10. 2022Understanding the effect of sparsity on neural networks robustnesscited by 8
  11. 2023Disentangling Linear Mode-Connectivitycited by 8
  12. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  13. 2022Decision boundaries and convex hulls in the feature space that deep learning functions learn from imagescited by 5
  14. 2022Explainable vision transformer enabled convolutional neural network for plant disease identification: PlantXViTcited by 5
  15. 2023Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networkscited by 5
  16. 2022Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networkscited by 4
  17. 2021Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?cited by 3
  18. 2021Programming with Neural Surrogates of Programscited by 3
  19. 2023Understanding MLP-Mixer as a Wide and Sparse MLPcited by 3
  20. 2023Towards model-free RL algorithms that scale well with unstructured datacited by 3
  21. 2021Adapting the Function Approximation Architecture in Online Reinforcement Learningcited by 2
  22. 2022MDMLP: Image Classification from Scratch on Small Datasets with MLPcited by 2
  23. 2023Weight-Sharing Regularizationcited by 2
  24. 2021The Effects of Image Distribution and Task on Adversarial Robustnesscited by 1
  25. 2022Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functionscited by 1
  26. 2024Artificial Human Intelligence: The role of Humans in the Development of Next Generation AIcited by 1
  27. 2024On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networkscited by 1
  28. 2025NM-Hebb: Coupling Local Hebbian Plasticity with Metric Learning for More Accurate and Interpretable CNNscited by 1
  29. 2022SplitMixer: Fat Trimmed From MLP-like Modelscited by 0
  30. 2022Studying inductive biases in image classification taskcited by 0
  31. 2023Subspace-Configurable Networkscited by 0
  32. 2023Unveiling Invariances via Neural Network Pruningcited by 0
  33. 2023Development of a Deep Learning Method to Identify Acute Ischemic Stroke Lesions on Brain CTcited by 0
  34. 2026The Computational Value of Sensory-Aligned Receptive Fields Depends on Neuronal Expressivitycited by 0