# Detecting out-of-distribution text using topological features of transformer-based language models

arXiv: https://arxiv.org/abs/2311.13102 · HTML: https://mlsanity.com/paper/2311.13102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Ripser: efficient computation of Vietoris-Rips persistence barcodes](https://mlsanity.com/paper/1908.02518/) (2019)
- [Types of Out-of-Distribution Texts and How to Detect Them](https://mlsanity.com/paper/2109.06827/) (2021)
- [Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection](https://mlsanity.com/paper/2101.03778/) (2021)
- [Artificial Text Detection via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2109.04825/) (2021)
- [News Category Dataset](https://mlsanity.com/paper/2209.11429/) (2022)
- [Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection](https://mlsanity.com/paper/2305.13282/) (2023)
- [The Topological BERT: Transforming Attention into Topology for Natural Language Processing](https://mlsanity.com/paper/2206.15195/) (2022)
- [Beyond Mahalanobis-Based Scores for Textual OOD Detection](https://mlsanity.com/paper/2211.13527/) (2022)
- [MLGuard: Defend Your Machine Learning Model!](https://mlsanity.com/paper/2309.01379/) (2023)

## Built on by (most cited first, 2 of 2)

- [Topological Data Analysis Applications in Natural Language Processing: A Survey](https://mlsanity.com/paper/2411.10298/) (2024)
- [An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face](https://mlsanity.com/paper/2512.07983/) (2025)

## Abstract (arXiv)

To safeguard machine learning systems that operate on textual data against out-of-distribution (OOD) inputs that could cause unpredictable behaviour, we explore the use of topological features of self-attention maps from transformer-based language models to detect when input text is out of distribution. Self-attention forms the core of transformer-based language models, dynamically assigning vectors to words based on context, thus in theory our methodology is applicable to any transformer-based language model with multihead self-attention. We evaluate our approach on BERT and compare it to a traditional OOD approach using CLS embeddings. Our results show that our approach outperforms CLS embeddings in distinguishing in-distribution samples from far-out-of-domain samples, but struggles with near or same-domain datasets.