mlsanityData to 7 Oct 2026

How far can we go without convolution: Improving fully-connected networks

arXiv 1511.02580 ↗First posted 9 Nov 2015cs.LG cs.NEcited by 20

Not on the map. Why?

Abstract (arXiv)

We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is just 10% short of a decent convolutional network.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 2; 2023: 5; 2024: 6; 2025: 3.

20 papers.

Sort:

By area:

  1. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  2. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  3. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  4. 2023Polynomial Time Cryptanalytic Extraction of Neural Network Modelscited by 17
  5. 2023Emergent representations in networks trained with the Forward-Forward algorithmcited by 11
  6. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  7. 2024The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networkscited by 8
  8. 2024Polynomial Time Cryptanalytic Extraction of Deep Neural Networks in the Hard-Label Settingcited by 8
  9. 2023Model-agnostic Measure of Generalization Difficultycited by 7
  10. 2024Towards Exact Computation of Inductive Biascited by 5
  11. 2024On Training of Kolmogorov-Arnold Networkscited by 4
  12. 2024Spectral GNN via Two-dimensional (2-D) Graph Convolutioncited by 3
8 more
  1. 2022MDMLP: Image Classification from Scratch on Small Datasets with MLPcited by 2
  2. 2022Deep Neural Networks as Complex Networkscited by 2
  3. 2021Accelerating Fully Connected Neural Network on Optical Network-on-Chip (ONoC)cited by 0
  4. 2021Cascaded Classifier for Pareto-Optimal Accuracy-Cost Trade-Off Using off-the-Shelf ANNscited by 0
  5. 2021SPINE: Soft Piecewise Interpretable Neural Equationscited by 0
  6. 2025Artificial Intelligence and Deep Learning Algorithms for Epigenetic Sequence Analysis: A Review for Epigeneticists and AI Expertscited by 0
  7. 2025A Simple Baseline for Stable and Plastic Neural Networkscited by 0
  8. 2025Modest-Align: Data-Efficient Alignment for Vision-Language Modelscited by 0