# Layer-wise training of deep networks using kernel similarity

arXiv: https://arxiv.org/abs/1703.07115 · HTML: https://mlsanity.com/paper/1703.07115/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2017.

## Built on by (most cited first, 15 of 15)

- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Feature Learning and Signal Propagation in Deep Neural Networks](https://mlsanity.com/paper/2110.11749/) (2021)
- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Module-wise Training of Neural Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2309.17357/) (2023)
- [Seeking Interpretability and Explainability in Binary Activated Neural Networks](https://mlsanity.com/paper/2209.03450/) (2022)
- [Deep Layer-wise Networks Have Closed-Form Weights](https://mlsanity.com/paper/2202.01210/) (2022)
- [An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture](https://mlsanity.com/paper/2211.06860/) (2022)
- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [Block-wise Training of Residual Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2210.00949/) (2022)
- [Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networks](https://mlsanity.com/paper/2411.17201/) (2024)
- [Convolutional Neural Networks Exploiting Attributes of Biological Neurons](https://mlsanity.com/paper/2311.08314/) (2023)
- [Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning](https://mlsanity.com/paper/2507.11367/) (2025)

## Abstract (arXiv)

Deep learning has shown promising results in many machine learning applications. The hierarchical feature representation built by deep networks enable compact and precise encoding of the data. A kernel analysis of the trained deep networks demonstrated that with deeper layers, more simple and more accurate data representations are obtained. In this paper, we propose an approach for layer-wise training of a deep network for the supervised classification task. A transformation matrix of each layer is obtained by solving an optimization aimed at a better representation where a subsequent layer builds its representation on the top of the features produced by a previous layer. We compared the performance of our approach with a DNN trained using back-propagation which has same architecture as ours. Experimental results on the real image datasets demonstrate efficacy of our approach. We also performed kernel analysis of layer representations to validate the claim of better feature encoding.