# GePpeTto Carves Italian into a Language Model

arXiv: https://arxiv.org/abs/2004.14253 · HTML: https://mlsanity.com/paper/2004.14253/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [IT5: Text-to-text Pretraining for Italian Language Understanding and Generation](https://mlsanity.com/paper/2203.03759/) (2022)
- [Fauno: The Italian Large Language Model that will leave you senza parole!](https://mlsanity.com/paper/2306.14457/) (2023)
- [Camoscio: an Italian Instruction-tuned LLaMA](https://mlsanity.com/paper/2307.16456/) (2023)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?](https://mlsanity.com/paper/2305.15908/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark](https://mlsanity.com/paper/2406.17535/) (2024)
- [Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation](https://mlsanity.com/paper/2311.15698/) (2023)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns?](https://mlsanity.com/paper/2208.14554/) (2022)
- [Understanding Emotion Valence is a Joint Deep Learning Task](https://mlsanity.com/paper/2305.17422/) (2023)
- [Predictability and Causality in Spanish and English Natural Language Generation](https://mlsanity.com/paper/2408.14283/) (2024)
- [Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English](https://mlsanity.com/paper/2504.09886/) (2025)
- [Charting a Decade of Computational Linguistics in Italy: The CLiC-it Corpus](https://mlsanity.com/paper/2509.19033/) (2025)
- [Language, Language Models, and What We're Talking About](https://mlsanity.com/paper/2609.03577/) (2026)

## Abstract (arXiv)

In the last few years, pre-trained neural architectures have provided impressive improvements across several NLP tasks. Still, generative language models are available mainly for English. We develop GePpeTto, the first generative language model for Italian, built using the GPT-2 architecture. We provide a thorough analysis of GePpeTto's quality by means of both an automatic and a human-based evaluation. The automatic assessment consists in (i) calculating perplexity across different genres and (ii) a profiling analysis over GePpeTto's writing characteristics. We find that GePpeTto's production is a sort of bonsai version of human production, with shorter but yet complex sentences. Human evaluation is performed over a sentence completion task, where GePpeTto's output is judged as natural more often than not, and much closer to the original human texts than to a simpler language model which we take as baseline.