# Evalita-LLM: Benchmarking Large Language Models on Italian

arXiv: https://arxiv.org/abs/2502.02289 · HTML: https://mlsanity.com/paper/2502.02289/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA](https://mlsanity.com/paper/2405.07101/) (2024)
- [A NotSo Simple Way to Beat Simple Bench](https://mlsanity.com/paper/2412.12173/) (2024)

## Built on by (most cited first, 7 of 7)

- [All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark](https://mlsanity.com/paper/2502.16989/) (2025)
- [An Empirical Investigation of Gender Stereotype Representation in Large Language Models: The Italian Case](https://mlsanity.com/paper/2507.19156/) (2025)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models](https://mlsanity.com/paper/2509.25922/) (2025)
- [Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM](https://mlsanity.com/paper/2512.21580/) (2025)
- [Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite](https://mlsanity.com/paper/2604.01957/) (2026)
- [Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs](https://mlsanity.com/paper/2605.07731/) (2026)

## Abstract (arXiv)

We describe Evalita-LLM, a new benchmark designed to evaluate Large Language Models (LLMs) on Italian tasks. The distinguishing and innovative features of Evalita-LLM are the following: (i) all tasks are native Italian, avoiding issues of translating from Italian and potential cultural biases; (ii) in addition to well established multiple-choice tasks, the benchmark includes generative tasks, enabling more natural interaction with LLMs; (iii) all tasks are evaluated against multiple prompts, this way mitigating the model sensitivity to specific prompts and allowing a fairer and objective evaluation. We propose an iterative methodology, where candidate tasks and candidate prompts are validated against a set of LLMs used for development. We report experimental results from the benchmark's development phase, and provide performance statistics for several state-of-the-art LLMs.