# Discriminative Recurrent Sparse Auto-Encoders

arXiv: https://arxiv.org/abs/1301.3775 · HTML: https://mlsanity.com/paper/1301.3775/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2013.

## Built on by (most cited first, 5 of 5)

- [You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences](https://mlsanity.com/paper/2606.15956/) (2026)
- [Supervised Dimensionality Reduction and Image Classification Utilizing Convolutional Autoencoders](https://mlsanity.com/paper/2208.12152/) (2022)
- [Sparse, Geometric Autoencoder Models of V1](https://mlsanity.com/paper/2302.11162/) (2023)
- [Learning Linear Groups in Neural Networks](https://mlsanity.com/paper/2305.18552/) (2023)
- [An NMF-Based Building Block for Interpretable Neural Networks With Continual Learning](https://mlsanity.com/paper/2311.11485/) (2023)

## Abstract (arXiv)

We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit all the power of deep networks, while substantially reducing the number of trainable parameters. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.