# Learning a Word-Level Language Model with Sentence-Level Noise Contrastive Estimation for Contextual Sentence Probability Estimation

arXiv: https://arxiv.org/abs/2103.07875 · HTML: https://mlsanity.com/paper/2103.07875/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 14 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://mlsanity.com/paper/1611.01462/) (2016)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [Frustratingly Short Attention Spans in Neural Language Modeling](https://mlsanity.com/paper/1702.04521/) (2017)
- [Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series](https://mlsanity.com/paper/1504.01575/) (2015)
- [Neural Lattice Language Models](https://mlsanity.com/paper/1803.05071/) (2018)
- [Learning neural trans-dimensional random field language models with noise-contrastive estimation](https://mlsanity.com/paper/1710.10739/) (2017)
- [Multiscale sequence modeling with a learned dictionary](https://mlsanity.com/paper/1707.00762/) (2017)
- [Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation](https://mlsanity.com/paper/1807.00993/) (2018)
- [On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation](https://mlsanity.com/paper/1602.06064/) (2016)
- [A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models](https://mlsanity.com/paper/1708.05997/) (2017)

## Abstract (arXiv)

Inferring the probability distribution of sentences or word sequences is a key process in natural language processing. While word-level language models (LMs) have been widely adopted for computing the joint probabilities of word sequences, they have difficulty in capturing a context long enough for sentence probability estimation (SPE). To overcome this, recent studies introduced training methods using sentence-level noise-contrastive estimation (NCE) with recurrent neural networks (RNNs). In this work, we attempt to extend it for contextual SPE, which aims to estimate a conditional sentence probability given a previous text. The proposed NCE samples negative sentences independently of a previous text so that the trained model gives higher probabilities to the sentences that are more consistent with textcolorbluethe context. We apply our method to a simple word-level RNN LM to focus on the effect of the sentence-level NCE training rather than on the network architecture. The quality of estimation was evaluated against multiple-choice cloze-style questions including both human and automatically generated questions. The experimental results show that the proposed method improved the SPE quality for the word-level RNN LM.