# Understanding and Training Deep Diagonal Circulant Neural Networks

arXiv: https://arxiv.org/abs/1901.10255 · HTML: https://mlsanity.com/paper/1901.10255/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 4 of 4)

- [CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices](https://mlsanity.com/paper/2510.25323/) (2025)
- [A Structured Sparse Neural Network and Its Matrix Calculations Algorithm](https://mlsanity.com/paper/2207.00903/) (2022)
- [Compact Circulant Layers with Spectral Priors](https://mlsanity.com/paper/2602.21965/) (2026)
- [Two-Valued Symmetric Circulant Matrices: Applications in Deep Learning](https://mlsanity.com/paper/2605.16443/) (2026)

## Abstract (arXiv)

In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones. Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks. Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers. We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models. We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach. Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.