# Interpretable Predictability-Based AI Text Detection: A Replication Study

arXiv: https://arxiv.org/abs/2603.15034 · HTML: https://mlsanity.com/paper/2603.15034/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning](https://mlsanity.com/paper/2410.20964/) (2024)
- [Few-Shot Detection of Machine-Generated Text using Style Representations](https://mlsanity.com/paper/2401.06712/) (2024)
- [Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian](https://mlsanity.com/paper/2206.01583/) (2022)
- [Robust AI-Generated Text Detection by Restricted Embeddings](https://mlsanity.com/paper/2410.08113/) (2024)
- [IMGTB: A Framework for Machine-Generated Text Detection Benchmarking](https://mlsanity.com/paper/2311.12574/) (2023)
- [Authorship Attribution in Multilingual Machine-Generated Texts](https://mlsanity.com/paper/2508.01656/) (2025)
- [Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings](https://mlsanity.com/paper/2501.18998/) (2025)
- [Classification of Human- and AI-Generated Texts for English, French, German, and Spanish](https://mlsanity.com/paper/2312.04882/) (2023)
- [PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?](https://mlsanity.com/paper/2404.05483/) (2024)
- [TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs](https://mlsanity.com/paper/2407.04981/) (2024)

## Abstract (arXiv)

This paper replicates and extends the system used in the AuTexTification shared task for authorship attribution of machine-generated texts. Exact replication was not possible because of differences in data splits, model availability, and implementation details, which we document as a case study in reproducibility. We tested newer multilingual language models (mDeBERTa-v3-base, Qwen, mGPT) and added 26 document-level stylometric features, using ablation, permutation importance, and SHAP analysis to assess feature influence. A single shared configuration was applied to both English and Spanish across Subtask 1 and Subtask 2. Averaged over three random seeds, the shared multilingual configuration performs comparably to or better than the language-specific baseline, with the clearest gains on model attribution (Subtask 2). The additional stylometric features yield small improvements, led by lexical diversity, but their contribution falls within seed variance once predictability-based probabilities are included, which remain the dominant signal. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.