# Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset

arXiv: https://arxiv.org/abs/2605.29365 · HTML: https://mlsanity.com/paper/2605.29365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 12 of 12)

- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [Balancing out Bias: Achieving Fairness Through Balanced Training](https://mlsanity.com/paper/2109.08253/) (2021)
- [Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction](https://mlsanity.com/paper/2404.14215/) (2024)
- [Style-Specific Neurons for Steering LLMs in Text Style Transfer](https://mlsanity.com/paper/2410.00593/) (2024)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer](https://mlsanity.com/paper/2309.08583/) (2023)
- [Text Style Transfer: An Introductory Overview](https://mlsanity.com/paper/2407.14822/) (2024)
- [Steering Large Language Models with Register Analysis for Arbitrary Style Transfer](https://mlsanity.com/paper/2505.00679/) (2025)
- [On Positivity Bias in Negative Reviews](https://mlsanity.com/paper/2106.12056/) (2021)
- [Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics](https://mlsanity.com/paper/2502.15022/) (2025)

## Abstract (arXiv)

Formality transfer is commonly framed as a symmetric bidirectional task between informal and formal registers. We argue that this framing conceals a supervision design flaw in existing benchmarks such as GYAFC: binary human rewrites encode relative stylistic shifts rather than absolute human notions of formality. Consequently, models learn to generate pseudo-formal outputs that satisfy benchmark labels while failing to produce genuinely formal language. We quantify this misalignment by re-evaluating benchmark formal labels under a human-aligned definition of formality, revealing substantial discrepancies that propagate to consistent informal-to-formal failures across model families. To address this issue, we reconceptualize formality transfer as a graded dimension rather than a binary attribute. We introduce a three-level spectrum: informal, casual, and formal, where casual serves as an explicit intermediate state that clarifies supervision signals. Based on this framework, we introduce 3LF, a dataset providing parallel supervision across all three levels. Training on 3LF substantially reduces informal-to-formal failures and improves alignment with human perception. For example, GPT-4.1-nano improves from 0.06 to 0.88 F1 in the informal-to-formal direction despite 3LF being significantly smaller than GYAFC. We further demonstrate that these gains cannot be reproduced through in-context learning alone and provide qualitative analyses of ambiguity-driven errors and meaning distortions. Overall, our findings demonstrate how supervision design shapes stylistic alignment and highlight the importance of alignment-aware benchmark construction in controllable text generation.