# Fine-tuning GPT-3 for Russian Text Summarization

arXiv: https://arxiv.org/abs/2108.03502 · HTML: https://mlsanity.com/paper/2108.03502/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2](https://mlsanity.com/paper/2006.01997/) (2020)
- [Self-Attentive Model for Headline Generation](https://mlsanity.com/paper/1901.07786/) (2019)
- [Dataset for Automatic Summarization of Russian News](https://mlsanity.com/paper/2006.11063/) (2020)
- [Advances of Transformer-Based Models for News Headline Generation](https://mlsanity.com/paper/2007.05044/) (2020)
- [Automatic Speech Summarisation: A Scoping Review](https://mlsanity.com/paper/2008.11897/) (2020)

## Built on by (most cited first, 12 of 12)

- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [On the Multilingual Capabilities of Very Large-Scale English Language Models](https://mlsanity.com/paper/2108.13349/) (2021)
- [A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions](https://mlsanity.com/paper/2307.05638/) (2023)
- [CloChat: Understanding How People Customize, Interact, and Experience Personas in Large Language Models](https://mlsanity.com/paper/2402.15265/) (2024)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?](https://mlsanity.com/paper/2411.05059/) (2024)
- [On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing](https://mlsanity.com/paper/2501.02191/) (2025)
- [PAGnol: An Extra-Large French Generative Model](https://mlsanity.com/paper/2110.08554/) (2021)
- [Automatic Summarization of Russian Texts: Comparison of Extractive and Abstractive Methods](https://mlsanity.com/paper/2206.09253/) (2022)
- [Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning](https://mlsanity.com/paper/2605.08765/) (2026)
- [Towards Better Understanding of Cybercrime: The Role of Fine-Tuned LLMs in Translation](https://mlsanity.com/paper/2404.01940/) (2024)
- [ProdRev: A DNN framework for empowering customers using generative pre-trained transformers](https://mlsanity.com/paper/2505.13491/) (2025)

## Abstract (arXiv)

Automatic summarization techniques aim to shorten and generalize information given in the text while preserving its core message and the most relevant ideas. This task can be approached and treated with a variety of methods, however, not many attempts have been made to produce solutions specifically for the Russian language despite existing localizations of the state-of-the-art models. In this paper, we aim to showcase ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. Additionally, we employ hyperparameter tuning so that the model's output becomes less random and more tied to the original text. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function. Despite being able to produce sensible summaries, our model still suffers from a number of flaws, namely, it is prone to altering Named Entities present in the original text (such as surnames, places, dates), deviating from facts stated in the given document, and repeating the information in the summary.