# Discrete Autoencoders for Sequence Models

arXiv: https://arxiv.org/abs/1801.09797 · HTML: https://mlsanity.com/paper/1801.09797/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2018.

## Built on by (most cited first, 24 of 24)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Sparse is Enough in Scaling Transformers](https://mlsanity.com/paper/2111.12763/) (2021)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [ScanFormer: Referring Expression Comprehension by Iteratively Scanning](https://mlsanity.com/paper/2406.18048/) (2024)
- [Dynamic Spatio-Temporal Specialization Learning for Fine-Grained Action Recognition](https://mlsanity.com/paper/2209.01425/) (2022)
- [Learning Gabor Texture Features for Fine-Grained Recognition](https://mlsanity.com/paper/2308.05396/) (2023)
- [RL for Reasoning by Adaptively Revealing Rationales](https://mlsanity.com/paper/2506.18110/) (2025)
- [Referring Expression Comprehension Using Language Adaptive Inference](https://mlsanity.com/paper/2306.04451/) (2023)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication](https://mlsanity.com/paper/2111.06464/) (2021)
- [DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models](https://mlsanity.com/paper/2411.03250/) (2024)
- [Hidden Schema Networks](https://mlsanity.com/paper/2207.03777/) (2022)
- [TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions](https://mlsanity.com/paper/2403.11818/) (2024)
- [GVDIFF: Grounded Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2407.01921/) (2024)
- [Large Language Model as Token Compressor and Decompressor](https://mlsanity.com/paper/2603.25340/) (2026)
- [Symbolic Autoencoding for Self-Supervised Sequence Learning](https://mlsanity.com/paper/2402.10575/) (2024)
- [DecisioNet: A Binary-Tree Structured Neural Network](https://mlsanity.com/paper/2207.01127/) (2022)
- [On Designing Diffusion Autoencoders for Efficient Generation and Representation Learning](https://mlsanity.com/paper/2506.00136/) (2025)
- [Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generation](https://mlsanity.com/paper/2111.08453/) (2021)
- [Dual Learning for Large Vocabulary On-Device ASR](https://mlsanity.com/paper/2301.04327/) (2023)
- [Generative Model for Models: Rapid DNN Customization for Diverse Tasks and Resource Constraints](https://mlsanity.com/paper/2308.15003/) (2023)
- [Learning Binary Autoencoder-Based Codes with Progressive Training](https://mlsanity.com/paper/2511.09221/) (2025)
- [Spatial-Temporal Expert Learning for Video-based Person Re-identification](https://mlsanity.com/paper/2607.01353/) (2026)
- [Quasi-Binarized Autoencoders: An Architecture-Independent Information Bottleneck for Medical Image Anomaly Detection](https://mlsanity.com/paper/2610.09670/) (2026)

## Abstract (arXiv)

Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.