# Robust AI-Generated Text Detection by Restricted Embeddings

arXiv: https://arxiv.org/abs/2410.08113 · HTML: https://mlsanity.com/paper/2410.08113/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://mlsanity.com/paper/2106.09022/) (2021)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection](https://mlsanity.com/paper/2101.03778/) (2021)
- [Outliers Dimensions that Disrupt Transformers Are Driven by Frequency](https://mlsanity.com/paper/2205.11380/) (2022)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [The Low-Dimensional Linear Geometry of Contextualized Word Representations](https://mlsanity.com/paper/2105.07109/) (2021)
- [GPT-who: An Information Density-based Machine-Generated Text Detector](https://mlsanity.com/paper/2310.06202/) (2023)

## Built on by (most cited first, 18 of 18)

- [Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders](https://mlsanity.com/paper/2503.03601/) (2025)
- [DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning](https://mlsanity.com/paper/2510.17489/) (2025)
- [DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models](https://mlsanity.com/paper/2509.14268/) (2025)
- [Real, Fake, or Manipulated? Detecting Machine-Influenced Text](https://mlsanity.com/paper/2509.15350/) (2025)
- [A Lightweight Approach to Detection of AI-Generated Texts Using Stylometric Features](https://mlsanity.com/paper/2511.21744/) (2025)
- [Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection](https://mlsanity.com/paper/2507.15286/) (2025)
- [Advacheck at GenAI Detection Task 1: AI Detection Powered by Domain-Aware Multi-Tasking](https://mlsanity.com/paper/2411.11736/) (2024)
- [Synthetic Tabular Data Detection In the Wild](https://mlsanity.com/paper/2503.01937/) (2025)
- [Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story](https://mlsanity.com/paper/2511.15210/) (2025)
- [Cross-table Synthetic Tabular Data Detection](https://mlsanity.com/paper/2412.13227/) (2024)
- [Detecting AI-Generated Paraphrases in Bengali: A Comparative Study of Zero-Shot and Fine-Tuned Transformers](https://mlsanity.com/paper/2512.21709/) (2025)
- [Interpretable Predictability-Based AI Text Detection: A Replication Study](https://mlsanity.com/paper/2603.15034/) (2026)
- [Amplifying, Not Learning: The Price of Out-of-Distribution Generalization in AI-Text Detection](https://mlsanity.com/paper/2605.21653/) (2026)
- [SV-Detect: AI-generated Text Detection with Steering Vectors](https://mlsanity.com/paper/2606.07313/) (2026)
- [MD-ProTector: Positioning Multiple Data-Driven Prototypes for LLM-Generated Text Detection](https://mlsanity.com/paper/2608.10459/) (2026)
- [How Much Were You Told? Measuring External Information in Peer Reviews](https://mlsanity.com/paper/2609.28041/) (2026)
- [A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID](https://mlsanity.com/paper/2609.30287/) (2026)
- [C-HAT-Bench: Benchmarking Chinese AI-Text Detection Beyond Fully Generated Text](https://mlsanity.com/paper/2609.32770/) (2026)

## Abstract (arXiv)

Growing amount and quality of AI-generated texts makes detecting such content more difficult. In most real-world scenarios, the domain (style and topic) of generated data and the generator model are not known in advance. In this work, we focus on the robustness of classifier-based detectors of AI-generated text, namely their ability to transfer to unseen generators or semantic domains. We investigate the geometry of the embedding space of Transformer-based text encoders and show that clearing out harmful linear subspaces helps to train a robust classifier, ignoring domain-specific spurious features. We investigate several subspace decomposition and feature selection strategies and achieve significant improvements over state of the art methods in cross-domain and cross-generator transfer. Our best approaches for head-wise and coordinate-based subspace removal increase the mean out-of-distribution (OOD) classification score by up to 9% and 14% in particular setups for RoBERTa and BERT embeddings respectively. We release our code and data: https://github.com/SilverSolver/RobustATD