# Understanding writing style in social media with a supervised contrastively pre-trained transformer arXiv: https://arxiv.org/abs/2310.11081 ยท HTML: https://mlsanity.com/paper/2310.11081/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 64 references that are in the map. - First posted on arXiv 17 Oct 2023. ## Builds on (most cited first, 22 of 22) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020) - [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021) - [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022) - [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023) - [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018) - [Encoding Musical Style with Transformer Autoencoders](https://mlsanity.com/paper/1912.05537/) (2019) - [Styleformer: Transformer based Generative Adversarial Networks with Style Vector](https://mlsanity.com/paper/2106.07023/) (2021) - [Task-Guided and Path-Augmented Heterogeneous Network Embedding for Author Identification](https://mlsanity.com/paper/1612.02814/) (2016) - [The Topic Confusion Task: A Novel Scenario for Authorship Attribution](https://mlsanity.com/paper/2104.08530/) (2021) - [Whodunit? Learning to Contrast for Authorship Attribution](https://mlsanity.com/paper/2209.11887/) (2022) - [Pair-Level Supervised Contrastive Learning for Natural Language Inference](https://mlsanity.com/paper/2201.10927/) (2022) - [DeepStyle: User Style Embedding for Authorship Attribution of Short Texts](https://mlsanity.com/paper/2103.11798/) (2021) - [A Self-supervised Representation Learning of Sentence Structure for Authorship Attribution](https://mlsanity.com/paper/2010.06786/) (2020) - [Unified and Multilingual Author Profiling for Detecting Haters](https://mlsanity.com/paper/2109.09233/) (2021) - [Can You Fool AI by Doing a 180? unicodex2013 A Case Study on Authorship Analysis of Texts by Arata Osada](https://mlsanity.com/paper/2207.09085/) (2022) ## Built on by (most cited first, 12 of 12) - [Trends and Challenges in Authorship Analysis: A Review of ML, DL, and LLM Approaches](https://mlsanity.com/paper/2505.15422/) (2025) - [One-shot Style Transfer LLM log-probabilities for Authorship Attribution and Verification](https://mlsanity.com/paper/2510.13302/) (2025) - [AuthorMix: Modular Authorship Style Transfer via Layer-wise Adapter Mixing](https://mlsanity.com/paper/2603.23069/) (2026) - [Incentives shape how humans co-create with generative AI](https://mlsanity.com/paper/2604.03529/) (2026) - [DisTrack: a new Tool for Semi-automatic Misinformation Tracking in Online Social Networks](https://mlsanity.com/paper/2408.00633/) (2024) - [Estimating the Influence of Sequentially Correlated Literary Properties in Textual Classification: A Data-Centric Hypothesis-Testing Approach](https://mlsanity.com/paper/2411.04950/) (2024) - [An Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data](https://mlsanity.com/paper/2503.07303/) (2025) - [Authorship Impersonation via LLM Prompting does not Evade Authorship Verification Methods](https://mlsanity.com/paper/2603.29454/) (2026) - [Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest](https://mlsanity.com/paper/2604.18955/) (2026) - [Unsupervised Style Representation Learning for AI-Text Detection via Paraphrase Inversion](https://mlsanity.com/paper/2606.10099/) (2026) - [STEB: Style Text Embedding Benchmark](https://mlsanity.com/paper/2606.31741/) (2026) - [Predicting Steering Vectors and Adapter Weights for Few-Shot Author-Style Transfer](https://mlsanity.com/paper/2610.03163/) (2026) ## Abstract (arXiv) Online Social Networks serve as fertile ground for harmful behavior, ranging from hate speech to the dissemination of disinformation. Malicious actors now have unprecedented freedom to misbehave, leading to severe societal unrest and dire consequences, as exemplified by events such as the Capitol assault during the US presidential election and the Antivaxx movement during the COVID-19 pandemic. Understanding online language has become more pressing than ever. While existing works predominantly focus on content analysis, we aim to shift the focus towards understanding harmful behaviors by relating content to their respective authors. Numerous novel approaches attempt to learn the stylistic features of authors in texts, but many of these approaches are constrained by small datasets or sub-optimal training losses. To overcome these limitations, we introduce the Style Transformer for Authorship Representations (STAR), trained on a large corpus derived from public sources of 4.5 x 106 authored texts involving 70k heterogeneous authors. Our model leverages Supervised Contrastive Loss to teach the model to minimize the distance between texts authored by the same individual. This author pretext pre-training task yields competitive performance at zero-shot with PAN challenges on attribution and clustering. Additionally, we attain promising results on PAN verification challenges using a single dense layer, with our model serving as an embedding encoder. Finally, we present results from our test partition on Reddit. Using a support base of 8 documents of 512 tokens, we can discern authors from sets of up to 1616 authors with at least 80% accuracy. We share our pre-trained model at huggingface (https://huggingface.co/AIDA-UPM/star) and our code is available at (https://github.com/jahuerta92/star)