# Document Context Language Models

arXiv: https://arxiv.org/abs/1511.03962 · HTML: https://mlsanity.com/paper/1511.03962/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2015.

## Built on by (most cited first, 10 of 10)

- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages](https://mlsanity.com/paper/2310.07418/) (2023)
- [A Survey of Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2203.02982/) (2022)
- [CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals](https://mlsanity.com/paper/2203.08774/) (2022)
- [Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism](https://mlsanity.com/paper/2207.00883/) (2022)
- [Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations](https://mlsanity.com/paper/2310.15815/) (2023)
- [The Blessings of Multiple Treatments and Outcomes in Treatment Effect Estimation](https://mlsanity.com/paper/2309.17283/) (2023)
- [SYRAC: Synthesize, Rank, and Count](https://mlsanity.com/paper/2310.01662/) (2023)
- [Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphs](https://mlsanity.com/paper/2310.10791/) (2023)
- [Symmetry-preserving graph attention network to solve routing problems at multiple resolutions](https://mlsanity.com/paper/2310.15543/) (2023)

## Abstract (arXiv)

Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.