# Generalizing and Hybridizing Count-based and Neural Language Models

arXiv: https://arxiv.org/abs/1606.00499 · HTML: https://mlsanity.com/paper/1606.00499/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2016.

## Built on by (most cited first, 11 of 11)

- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [The Role of n-gram Smoothing in the Age of Neural Networks](https://mlsanity.com/paper/2403.17240/) (2024)
- [NGM: A Plug-and-Play Training-Free Memory Module for LLMs](https://mlsanity.com/paper/2605.16893/) (2026)
- [Model-based Subsampling for Knowledge Graph Completion](https://mlsanity.com/paper/2309.09296/) (2023)
- [Efficient Training of Language Models with Compact and Consistent Next Token Distributions](https://mlsanity.com/paper/2407.02819/) (2024)
- [Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding](https://mlsanity.com/paper/2407.04251/) (2024)
- [The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation](https://mlsanity.com/paper/2610.20456/) (2026)

## Abstract (arXiv)

Language models (LMs) are statistical models that calculate probabilities over sequences of words or other discrete symbols. Currently two major paradigms for language modeling exist: count-based n-gram models, which have advantages of scalability and test-time speed, and neural LMs, which often achieve superior modeling performance. We demonstrate how both varieties of models can be unified in a single modeling framework that defines a set of probability distributions over the vocabulary of words, and then dynamically calculates mixture weights over these distributions. This formulation allows us to create novel hybrid models that combine the desirable features of count-based and neural LMs, and experiments demonstrate the advantages of these approaches.