# Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

arXiv: https://arxiv.org/abs/2109.04817 · HTML: https://mlsanity.com/paper/2109.04817/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 78 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [Politeness Transfer: A Tag and Generate Approach](https://mlsanity.com/paper/2004.14257/) (2020)
- [Personalized Machine Translation: Preserving Original Author Traits](https://mlsanity.com/paper/1610.05461/) (2016)
- [Explainable Authorship Verification in Social Media via Attention-based Similarity Learning](https://mlsanity.com/paper/1910.08144/) (2019)
- [Computational Sociolinguistics: A Survey](https://mlsanity.com/paper/1508.07544/) (2015)
- [The Importance of Suppressing Domain Style in Authorship Analysis](https://mlsanity.com/paper/2005.14714/) (2020)
- [Learning Stylometric Representations for Authorship Analysis](https://mlsanity.com/paper/1606.01219/) (2016)
- [metric-learn: Metric Learning Algorithms in Python](https://mlsanity.com/paper/1908.04710/) (2019)
- [(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas](https://mlsanity.com/paper/1909.00098/) (2019)
- [Deep Bayes Factor Scoring for Authorship Verification](https://mlsanity.com/paper/2008.10105/) (2020)
- [Low-Level Linguistic Controls for Style Transfer and Content Preservation](https://mlsanity.com/paper/1911.03385/) (2019)
- [Unsupervised Learning of Style-sensitive Word Vectors](https://mlsanity.com/paper/1805.05581/) (2018)

## Built on by (most cited first, 17 of 17)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples](https://mlsanity.com/paper/2410.12757/) (2024)
- [Learning Interpretable Style Embeddings via Prompting LLMs](https://mlsanity.com/paper/2305.12696/) (2023)
- [A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Texts](https://mlsanity.com/paper/2311.08374/) (2023)
- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer](https://mlsanity.com/paper/2309.08583/) (2023)
- [Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing](https://mlsanity.com/paper/2510.04120/) (2025)
- [WordDecipher: Enhancing Digital Workspace Communication with Explainable AI for Non-native English Speakers](https://mlsanity.com/paper/2404.07005/) (2024)
- [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](https://mlsanity.com/paper/2309.12250/) (2023)
- [Authorship identification under domain shift: a survey of stylistic measures and learned author representations](https://mlsanity.com/paper/2310.00436/) (2023)
- [HALvest-Contrastive: Retrieval-Like Authorship Attribution with Patch-Level Late Interaction](https://mlsanity.com/paper/2407.20595/) (2024)
- [StoryLens: Preference-Aligned Story Rewriting via Context-Aware Narrative Enrichment](https://mlsanity.com/paper/2605.28073/) (2026)
- [iBERT: Interpretable Embeddings via Sense Decomposition](https://mlsanity.com/paper/2510.09882/) (2025)
- [Where Does Authorship Signal Emerge in Encoder-Based Language Models?](https://mlsanity.com/paper/2605.19908/) (2026)
- [Unsupervised Style Representation Learning for AI-Text Detection via Paraphrase Inversion](https://mlsanity.com/paper/2606.10099/) (2026)
- [STEB: Style Text Embedding Benchmark](https://mlsanity.com/paper/2606.31741/) (2026)

## Abstract (arXiv)

Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac'tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures.