# Text vectorization via transformer-based language models and n-gram perplexities

arXiv: https://arxiv.org/abs/2307.09255 · HTML: https://mlsanity.com/paper/2307.09255/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 6 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Built on by (most cited first, 2 of 2)

- [MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems](https://mlsanity.com/paper/2508.21307/) (2025)
- [An Evaluation of LLMs Inference on Popular Single-board Computers](https://mlsanity.com/paper/2511.07425/) (2025)

## Abstract (arXiv)

As the probability (and thus perplexity) of a text is calculated based on the product of the probabilities of individual tokens, it may happen that one unlikely token significantly reduces the probability (i.e., increase the perplexity) of some otherwise highly probable input, while potentially representing a simple typographical error. Also, given that perplexity is a scalar value that refers to the entire input, information about the probability distribution within it is lost in the calculation (a relatively good text that has one unlikely token and another text in which each token is equally likely they can have the same perplexity value), especially for longer texts. As an alternative to scalar perplexity this research proposes a simple algorithm used to calculate vector values based on n-gram perplexities within the input. Such representations consider the previously mentioned aspects, and instead of a unique value, the relative perplexity of each text token is calculated, and these values are combined into a single vector representing the input.