# ACDC: A Structured Efficient Linear Layer

arXiv: https://arxiv.org/abs/1511.05946 · HTML: https://mlsanity.com/paper/1511.05946/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2015.

## Built on by (most cited first, 19 of 19)

- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Rethinking Neural Operations for Diverse Tasks](https://mlsanity.com/paper/2103.15798/) (2021)
- [LookupFFN: Making Transformers Compute-lite for CPU inference](https://mlsanity.com/paper/2403.07221/) (2024)
- [GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09650/) (2021)
- [Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling](https://mlsanity.com/paper/2402.18508/) (2024)
- [VMonarch: Efficient Video Diffusion Transformers with Structured Attention](https://mlsanity.com/paper/2601.22275/) (2026)
- [Is the Number of Trainable Parameters All That Actually Matters?](https://mlsanity.com/paper/2109.11928/) (2021)
- [Block Circulant Adapter for Large Language Models](https://mlsanity.com/paper/2505.00582/) (2025)
- [Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers](https://mlsanity.com/paper/2406.16450/) (2024)
- [Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects](https://mlsanity.com/paper/2508.03222/) (2025)
- [CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices](https://mlsanity.com/paper/2510.25323/) (2025)
- [Compressing Structured Tensor Algebra](https://mlsanity.com/paper/2407.13726/) (2024)
- [Converting Transformers into DGNNs Form](https://mlsanity.com/paper/2502.00585/) (2025)
- [Once-for-All Channel Mixers (HYPERTINYPW): Generative Compression for TinyML](https://mlsanity.com/paper/2603.24916/) (2026)
- [Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count](https://mlsanity.com/paper/2605.08171/) (2026)

## Abstract (arXiv)

The linear layer is one of the most pervasive modules in deep learning representations. However, it requires O(N2) parameters and O(N2) operations. These costs can be prohibitive in mobile applications or prevent scaling in many domains. Here, we introduce a deep, differentiable, fully-connected neural network module composed of diagonal matrices of parameters, A and D, and the discrete cosine transform C. The core module, structured as mathbfACDC-1, has O(N) parameters and incurs O(N log N ) operations. We present theoretical results showing how deep cascades of ACDC layers approximate linear layers. ACDC is, however, a stand-alone module and can be used in combination with any other types of module. In our experiments, we show that it can indeed be successfully interleaved with ReLU modules in convolutional neural networks for image recognition. Our experiments also study critical factors in the training of these structured modules, including initialization and depth. Finally, this paper also provides a connection between structured linear transforms used in deep learning and the field of Fourier optics, illustrating how ACDC could in principle be implemented with lenses and diffractive elements.