# SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification

arXiv: https://arxiv.org/abs/2407.05449 · HTML: https://mlsanity.com/paper/2407.05449/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Builds on (most cited first, 5 of 5)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)

## Built on by (most cited first, 3 of 3)

- [SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotators](https://mlsanity.com/paper/2502.06394/) (2025)
- [Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification](https://mlsanity.com/paper/2507.15557/) (2025)
- [The Tatoxa System for Text Detoxification in Low-Resource Languages: The Case of Tatar](https://mlsanity.com/paper/2606.26015/) (2026)

## Abstract (arXiv)

This paper presents a solution for the Multilingual Text Detoxification task in the PAN-2024 competition of the SmurfCat team. Using data augmentation through machine translation and a special filtering procedure, we collected an additional multilingual parallel dataset for text detoxification. Using the obtained data, we fine-tuned several multilingual sequence-to-sequence models, such as mT0 and Aya, on a text detoxification task. We applied the ORPO alignment technique to the final model. Our final model has only 3.7 billion parameters and achieves state-of-the-art results for the Ukrainian language and near state-of-the-art results for other languages. In the competition, our team achieved first place in the automated evaluation with a score of 0.52 and second place in the final human evaluation with a score of 0.74.