# Oddballness: universal anomaly detection with language models

arXiv: https://arxiv.org/abs/2409.03046 · HTML: https://mlsanity.com/paper/2409.03046/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 7 of 7)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [LogGPT: Log Anomaly Detection via GPT](https://mlsanity.com/paper/2309.14482/) (2023)
- [Compositional Sequence Labeling Models for Error Detection in Learner Writing](https://mlsanity.com/paper/1607.06153/) (2016)
- [Context is Key: Grammatical Error Detection with Contextual Word Representations](https://mlsanity.com/paper/1906.06593/) (2019)
- [Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection](https://mlsanity.com/paper/1904.07334/) (2019)

## Built on by (most cited first, 2 of 2)

- [From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors](https://mlsanity.com/paper/2509.20065/) (2025)
- [Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts](https://mlsanity.com/paper/2505.04507/) (2025)

## Abstract (arXiv)

We present a new method to detect anomalies in texts (in general: in sequences of any data), using language models, in a totally unsupervised manner. The method considers probabilities (likelihoods) generated by a language model, but instead of focusing on low-likelihood tokens, it considers a new metric introduced in this paper: oddballness. Oddballness measures how ``strange'' a given token is according to the language model. We demonstrate in grammatical error detection tasks (a specific case of text anomaly detection) that oddballness is better than just considering low-likelihood events, if a totally unsupervised setup is assumed.