# Architectural Complexity Measures of Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1602.08210 · HTML: https://mlsanity.com/paper/1602.08210/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2016.

## Built on by (most cited first, 17 of 17)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation](https://mlsanity.com/paper/2110.11852/) (2021)
- [Recurrent Neural Networks for Dynamical Systems: Applications to Ordinary Differential Equations, Collective Motion, and Hydrological Modeling](https://mlsanity.com/paper/2202.07022/) (2022)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Do Transformer World Models Give Better Policy Gradients?](https://mlsanity.com/paper/2402.05290/) (2024)
- [TSFool: Crafting Highly-Imperceptible Adversarial Time Series through Multi-Objective Attack](https://mlsanity.com/paper/2209.06388/) (2022)
- [SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecasting](https://mlsanity.com/paper/2312.14880/) (2023)
- [Temporal Convolutions for Multi-Step Quadrotor Motion Prediction](https://mlsanity.com/paper/2110.04182/) (2021)
- [Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning](https://mlsanity.com/paper/2107.01081/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs](https://mlsanity.com/paper/2205.01940/) (2022)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Optimum Output Long Short-Term Memory Cell for High-Frequency Trading Forecasting](https://mlsanity.com/paper/2304.09840/) (2023)
- [The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulation](https://mlsanity.com/paper/2512.18607/) (2025)
- [Rethinking Recurrent Neural Networks for Time Series Forecasting: A Reinforced Recurrent Encoder with Prediction-Oriented Proximal Policy Optimization](https://mlsanity.com/paper/2601.03683/) (2026)

## Abstract (arXiv)

In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs). Our main contribution is twofold: first, we present a rigorous graph-theoretic framework describing the connecting architectures of RNNs in general. Second, we propose three architecture complexity measures of RNNs: (a) the recurrent depth, which captures the RNN's over-time nonlinear complexity, (b) the feedforward depth, which captures the local input-output nonlinearity (similar to the "depth" in feedforward neural networks (FNNs)), and (c) the recurrent skip coefficient which captures how rapidly the information propagates over time. We rigorously prove each measure's existence and computability. Our experimental results show that RNNs might benefit from larger recurrent depth and feedforward depth. We further demonstrate that increasing recurrent skip coefficient offers performance boosts on long term dependency problems.