# Lecture Notes: Neural Network Architectures

arXiv: https://arxiv.org/abs/2304.05133 · HTML: https://mlsanity.com/paper/2304.05133/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Fractional Deep Neural Network via Constrained Optimization](https://mlsanity.com/paper/2004.00719/) (2020)
- [Deep Neural Nets with Fixed Bias Configuration](https://mlsanity.com/paper/2107.01308/) (2021)
- [An Optimal Time Variable Learning Framework for Deep Neural Networks](https://mlsanity.com/paper/2204.08528/) (2022)

## Built on by (most cited first, 2 of 2)

- [How regularization affects the geometry of loss functions](https://mlsanity.com/paper/2307.15744/) (2023)
- [Time Regularization in Optimal Time Variable Learning](https://mlsanity.com/paper/2306.16111/) (2023)

## Abstract (arXiv)

These lecture notes provide an overview of Neural Network architectures from a mathematical point of view. Especially, Machine Learning with Neural Networks is seen as an optimization problem. Covered are an introduction to Neural Networks and the following architectures: Feedforward Neural Network, Convolutional Neural Network, ResNet, and Recurrent Neural Network.