# PART: Pre-trained Authorship Representation Transformer

arXiv: https://arxiv.org/abs/2209.15373 · HTML: https://mlsanity.com/paper/2209.15373/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 77 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [A Stylometric Inquiry into Hyperpartisan and Fake News](https://mlsanity.com/paper/1702.05638/) (2017)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start](https://mlsanity.com/paper/2010.02584/) (2020)
- [Encoding Musical Style with Transformer Autoencoders](https://mlsanity.com/paper/1912.05537/) (2019)
- [Styleformer: Transformer based Generative Adversarial Networks with Style Vector](https://mlsanity.com/paper/2106.07023/) (2021)
- [Task-Guided and Path-Augmented Heterogeneous Network Embedding for Author Identification](https://mlsanity.com/paper/1612.02814/) (2016)
- [Pair-Level Supervised Contrastive Learning for Natural Language Inference](https://mlsanity.com/paper/2201.10927/) (2022)
- [DeepStyle: User Style Embedding for Authorship Attribution of Short Texts](https://mlsanity.com/paper/2103.11798/) (2021)
- [A Self-supervised Representation Learning of Sentence Structure for Authorship Attribution](https://mlsanity.com/paper/2010.06786/) (2020)
- [Unified and Multilingual Author Profiling for Detecting Haters](https://mlsanity.com/paper/2109.09233/) (2021)
- [Zero and Few-shot Learning for Author Profiling](https://mlsanity.com/paper/2204.10543/) (2022)
- [Can You Fool AI by Doing a 180? unicodex2013 A Case Study on Authorship Analysis of Texts by Arata Osada](https://mlsanity.com/paper/2207.09085/) (2022)

## Built on by (most cited first, 13 of 13)

- [Can Large Language Models Identify Authorship?](https://mlsanity.com/paper/2403.08213/) (2024)
- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin](https://mlsanity.com/paper/2410.09245/) (2024)
- [Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?](https://mlsanity.com/paper/2311.07564/) (2023)
- [AIDBench: A benchmark for evaluating the authorship identification capability of large language models](https://mlsanity.com/paper/2411.13226/) (2024)
- [T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models](https://mlsanity.com/paper/2310.18454/) (2023)
- [Authorship identification under domain shift: a survey of stylistic measures and learned author representations](https://mlsanity.com/paper/2310.00436/) (2023)
- [Enhancing Authorship Attribution through Embedding Fusion: A Novel Approach with Masked and Encoder-Decoder Language Models](https://mlsanity.com/paper/2411.00411/) (2024)
- [Isolating authorship from content with semantic embeddings and contrastive learning](https://mlsanity.com/paper/2411.18472/) (2024)
- [BARD10: A New Benchmark Reveals Significance of Bangla Stop-Words in Authorship Attribution](https://mlsanity.com/paper/2511.08085/) (2025)
- [Foundational Study on Authorship Attribution of Japanese Web Reviews for Actor Analysis](https://mlsanity.com/paper/2604.16376/) (2026)
- [SWAN: Semantic Watermarking with Abstract Meaning Representation](https://mlsanity.com/paper/2605.04305/) (2026)
- [Privacy Personalization Trade offs in LLMs: The Impact of Stylometric Signal Reduction on User-Specific Text Generation](https://mlsanity.com/paper/2609.22112/) (2026)

## Abstract (arXiv)

Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. Using stylometric representations is more suitable, but this by itself is an open research challenge. In this paper, we propose PART, a contrastively trained model fit to learn authorship embeddings instead of semantics. We train our model on ~1.5M texts belonging to 1162 literature authors, 17287 blog posters and 135 corporate email accounts; a heterogeneous set with identifiable writing styles. We evaluate the model on current challenges, achieving competitive performance. We also evaluate our model on test splits of the datasets achieving zero-shot 72.39% accuracy when bounded to 250 authors, a 54% and 56% higher than RoBERTa embeddings. We qualitatively assess the representations with different data visualizations on the available datasets, observing features such as gender, age, or occupation of the author.