# Challenging Assumptions in Learning Generic Text Style Embeddings

arXiv: https://arxiv.org/abs/2501.16073 · HTML: https://mlsanity.com/paper/2501.16073/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Improving Disentangled Text Representation Learning with Information-Theoretic Guidance](https://mlsanity.com/paper/2006.00693/) (2020)
- [Few-Shot Detection of Machine-Generated Text using Style Representations](https://mlsanity.com/paper/2401.06712/) (2024)
- [StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples](https://mlsanity.com/paper/2410.12757/) (2024)
- [StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer](https://mlsanity.com/paper/2104.05196/) (2021)
- [Evaluating prose style transfer with the Bible](https://mlsanity.com/paper/1711.04731/) (2017)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)

## Abstract (arXiv)

Recent advancements in language representation learning primarily emphasize language modeling for deriving meaningful representations, often neglecting style-specific considerations. This study addresses this gap by creating generic, sentence-level style embeddings crucial for style-centric tasks. Our approach is grounded on the premise that low-level text style changes can compose any high-level style. We hypothesize that applying this concept to representation learning enables the development of versatile text style embeddings. By fine-tuning a general-purpose text encoder using contrastive learning and standard cross-entropy loss, we aim to capture these low-level style shifts, anticipating that they offer insights applicable to high-level text styles. The outcomes prompt us to reconsider the underlying assumptions as the results do not always show that the learned style representations capture high-level text styles.