# Forward Thinking: Building and Training Neural Networks One Layer at a Time

arXiv: https://arxiv.org/abs/1706.02480 · HTML: https://mlsanity.com/paper/1706.02480/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2017.

## Built on by (most cited first, 15 of 15)

- [OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses](https://mlsanity.com/paper/2204.02426/) (2022)
- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices](https://mlsanity.com/paper/2305.17005/) (2023)
- [How much pre-training is enough to discover a good subnetwork?](https://mlsanity.com/paper/2108.00259/) (2021)
- [An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture](https://mlsanity.com/paper/2211.06860/) (2022)
- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [Early-Exit Graph Neural Networks](https://mlsanity.com/paper/2505.18088/) (2025)
- [Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers](https://mlsanity.com/paper/2102.04697/) (2021)
- [Training Neural Networks Using Reproducing Kernel Space Interpolation and Model Reduction](https://mlsanity.com/paper/2308.16754/) (2023)
- [FADE: Enabling Federated Adversarial Training on Heterogeneous Resource-Constrained Edge Devices](https://mlsanity.com/paper/2209.03839/) (2022)
- [FedProphet: Memory-Efficient Federated Adversarial Training via Robust and Consistent Cascade Learning](https://mlsanity.com/paper/2409.08372/) (2024)
- [High-Throughput Detection of Risk Factors to Sudden Cardiac Arrest in Youth Athletes: A Smartwatch-Based Screening Platform](https://mlsanity.com/paper/2412.12118/) (2024)
- [Information flow in multilayer perceptrons: an in-depth analysis](https://mlsanity.com/paper/2510.13846/) (2025)
- [Motivation is Something You Need](https://mlsanity.com/paper/2602.21064/) (2026)
- [An optimal control approach for neural network architecture adaptation with a posteriori error estimation](https://mlsanity.com/paper/2607.07637/) (2026)

## Abstract (arXiv)

We present a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision trees. The main idea is that layers can be trained one at a time, and once they are trained, the input data are mapped forward through the layer to create a new learning problem. The process is repeated, transforming the data through multiple layers, one at a time, rendering a new data set, which is expected to be better behaved, and on which a final output layer can achieve good performance. We call this forward thinking and demonstrate a proof of concept by achieving state-of-the-art accuracy on the MNIST dataset for convolutional neural networks. We also provide a general mathematical formulation of forward thinking that allows for other types of deep learning problems to be considered.