# Direct Output Connection for a High-Rank Language Model

arXiv: https://arxiv.org/abs/1808.10143 · HTML: https://mlsanity.com/paper/1808.10143/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2018.

## Built on by (most cited first, 11 of 11)

- [Lessons on Parameter Sharing across Layers in Transformers](https://mlsanity.com/paper/2104.06022/) (2021)
- [Algorithmic progress in language models](https://mlsanity.com/paper/2403.05812/) (2024)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [Low-Resource Language Modelling of South African Languages](https://mlsanity.com/paper/2104.00772/) (2021)
- [Dependency-based Mixture Language Models](https://mlsanity.com/paper/2203.10256/) (2022)
- [Delving Deeper Into Astromorphic Transformers](https://mlsanity.com/paper/2312.10925/) (2023)
- [The Importance of the Current Input in Sequence Modeling](https://mlsanity.com/paper/2112.11776/) (2021)
- [Advancing State of the Art in Language Modeling](https://mlsanity.com/paper/2312.03735/) (2023)
- [Explicit Word Density Estimation for Language Modelling](https://mlsanity.com/paper/2406.10256/) (2024)
- [Copy-Paste Language Models](https://mlsanity.com/paper/2610.19247/) (2026)

## Abstract (arXiv)

This paper proposes a state-of-the-art recurrent neural network (RNN) language model that combines probability distributions computed not only from a final RNN layer but also from middle layers. Our proposed method raises the expressive power of a language model based on the matrix factorization interpretation of language modeling introduced by Yang et al. (2018). The proposed method improves the current state-of-the-art language model and achieves the best score on the Penn Treebank and WikiText-2, which are the standard benchmark datasets. Moreover, we indicate our proposed method contributes to two application tasks: machine translation and headline generation. Our code is publicly available at: https://github.com/nttcslab-nlp/doc_lm.