# Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors

arXiv: https://arxiv.org/abs/2505.24523 · HTML: https://mlsanity.com/paper/2505.24523/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2402.11175/) (2024)
- [Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective](https://mlsanity.com/paper/2210.10488/) (2022)
- [Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features](https://mlsanity.com/paper/2109.12258/) (2021)
- [SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection](https://mlsanity.com/paper/2404.14183/) (2024)
- [LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection](https://mlsanity.com/paper/2408.04284/) (2024)
- [Linguistic Profiling of a Neural Language Model](https://mlsanity.com/paper/2010.01869/) (2020)
- [Exploring the Limitations of Detecting Machine-Generated Text](https://mlsanity.com/paper/2406.11073/) (2024)
- [GenAI Content Detection Task 1: English and Multilingual Machine-Generated Text Detection: AI vs. Human](https://mlsanity.com/paper/2501.11012/) (2025)
- [AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italian](https://mlsanity.com/paper/2406.12128/) (2024)

## Built on by (most cited first, 9 of 9)

- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [DisarmRAG: Stealthy Retriever-Centric Poisoning to Disable Self-Correction in Retrieval-Augmented Generation (Extended Version)](https://mlsanity.com/paper/2508.20083/) (2025)
- [M-DAIGT: A Shared Task on Multi-Domain Detection of AI-Generated Text](https://mlsanity.com/paper/2511.11340/) (2025)
- [When New Generators Arrive: Lifelong Machine-Generated Text Attribution via Ridge Feature Transfer](https://mlsanity.com/paper/2606.05626/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection](https://mlsanity.com/paper/2511.01192/) (2025)
- [AEyeDE: An Attention-Based Attribution Framework for AI-Generated Text Detection](https://mlsanity.com/paper/2606.00016/) (2026)
- [Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts](https://mlsanity.com/paper/2608.09510/) (2026)
- [Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis](https://mlsanity.com/paper/2609.34691/) (2026)

## Abstract (arXiv)

Recent advancements in Generative AI and Large Language Models (LLMs) have enabled the creation of highly realistic synthetic content, raising concerns about the potential for malicious use, such as misinformation and manipulation. Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the lack of robust benchmarks that assess generalization to real-world scenarios. In this work, we present a pipeline to test the resilience of state-of-the-art MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed adversarial attacks. To challenge the detectors, we fine-tune language models using Direct Preference Optimization (DPO) to shift the MGT style toward human-written text (HWT). This exploits the detectors' reliance on stylistic clues, making new generations more challenging to detect. Additionally, we analyze the linguistic shifts induced by the alignment and which features are used by detectors to detect MGT texts. Our results show that detectors can be easily fooled with relatively few examples, resulting in a significant drop in detection performance. This highlights the importance of improving detection methods and making them robust to unseen in-domain texts.