# Alternative structures for character-level RNNs

arXiv: https://arxiv.org/abs/1511.06303 · HTML: https://mlsanity.com/paper/1511.06303/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2015.

## Built on by (most cited first, 5 of 5)

- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://mlsanity.com/paper/2102.01909/) (2021)
- [STAB: Speech Tokenizer Assessment Benchmark](https://mlsanity.com/paper/2409.02384/) (2024)
- [Adaptive-saturated RNN: Remember more with less instability](https://mlsanity.com/paper/2304.11790/) (2023)
- [Spoken Word2Vec: Learning Skipgram Embeddings from Speech](https://mlsanity.com/paper/2311.09319/) (2023)

## Abstract (arXiv)

Recurrent neural networks are convenient and efficient models for language modeling. However, when applied on the level of characters instead of words, they suffer from several problems. In order to successfully model long-term dependencies, the hidden representation needs to be large. This in turn implies higher computational costs, which can become prohibitive in practice. We propose two alternative structural modifications to the classical RNN model. The first one consists on conditioning the character level representation on the previous word representation. The other one uses the character history to condition the output probability. We evaluate the performance of the two proposed modifications on challenging, multi-lingual real world data.