# How far can we go without convolution: Improving fully-connected networks

arXiv: https://arxiv.org/abs/1511.02580 · HTML: https://mlsanity.com/paper/1511.02580/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2015.

## Built on by (most cited first, 20 of 20)

- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [Scaling MLPs: A Tale of Inductive Bias](https://mlsanity.com/paper/2306.13575/) (2023)
- [Polynomial Time Cryptanalytic Extraction of Neural Network Models](https://mlsanity.com/paper/2310.08708/) (2023)
- [Emergent representations in networks trained with the Forward-Forward algorithm](https://mlsanity.com/paper/2305.18353/) (2023)
- [Data-Efficient Multimodal Fusion on a Single GPU](https://mlsanity.com/paper/2312.10144/) (2023)
- [Polynomial Time Cryptanalytic Extraction of Deep Neural Networks in the Hard-Label Setting](https://mlsanity.com/paper/2410.05750/) (2024)
- [The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks](https://mlsanity.com/paper/2405.10928/) (2024)
- [Model-agnostic Measure of Generalization Difficulty](https://mlsanity.com/paper/2305.01034/) (2023)
- [Towards Exact Computation of Inductive Bias](https://mlsanity.com/paper/2406.15941/) (2024)
- [On Training of Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2411.05296/) (2024)
- [Spectral GNN via Two-dimensional (2-D) Graph Convolution](https://mlsanity.com/paper/2404.04559/) (2024)
- [MDMLP: Image Classification from Scratch on Small Datasets with MLP](https://mlsanity.com/paper/2205.14477/) (2022)
- [Deep Neural Networks as Complex Networks](https://mlsanity.com/paper/2209.05488/) (2022)
- [Accelerating Fully Connected Neural Network on Optical Network-on-Chip (ONoC)](https://mlsanity.com/paper/2109.14878/) (2021)
- [Cascaded Classifier for Pareto-Optimal Accuracy-Cost Trade-Off Using off-the-Shelf ANNs](https://mlsanity.com/paper/2110.14256/) (2021)
- [SPINE: Soft Piecewise Interpretable Neural Equations](https://mlsanity.com/paper/2111.10622/) (2021)
- [Artificial Intelligence and Deep Learning Algorithms for Epigenetic Sequence Analysis: A Review for Epigeneticists and AI Experts](https://mlsanity.com/paper/2504.03733/) (2025)
- [A Simple Baseline for Stable and Plastic Neural Networks](https://mlsanity.com/paper/2507.10637/) (2025)
- [Modest-Align: Data-Efficient Alignment for Vision-Language Models](https://mlsanity.com/paper/2510.21606/) (2025)

## Abstract (arXiv)

We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is just 10% short of a decent convolutional network.