# Fauno: The Italian Large Language Model that will leave you senza parole!

arXiv: https://arxiv.org/abs/2306.14457 · HTML: https://mlsanity.com/paper/2306.14457/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [Shape registration in the time of transformers](https://mlsanity.com/paper/2106.13679/) (2021)
- [A Survey on Efficient Training of Transformers](https://mlsanity.com/paper/2302.01107/) (2023)
- [IT5: Text-to-text Pretraining for Italian Language Understanding and Generation](https://mlsanity.com/paper/2203.03759/) (2022)
- [GePpeTto Carves Italian into a Language Model](https://mlsanity.com/paper/2004.14253/) (2020)
- [Multimodal Neural Databases](https://mlsanity.com/paper/2305.01447/) (2023)
- [CycleDRUMS: Automatic Drum Arrangement For Bass Lines Using CycleGAN](https://mlsanity.com/paper/2104.00353/) (2021)

## Built on by (most cited first, 13 of 13)

- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA](https://mlsanity.com/paper/2405.07101/) (2024)
- [Camoscio: an Italian Instruction-tuned LLaMA](https://mlsanity.com/paper/2307.16456/) (2023)
- [BgGPT 1.0: Extending English-centric LLMs to other languages](https://mlsanity.com/paper/2412.10893/) (2024)
- [RRAML: Reinforced Retrieval Augmented Machine Learning](https://mlsanity.com/paper/2307.12798/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark](https://mlsanity.com/paper/2406.17535/) (2024)
- [The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italian](https://mlsanity.com/paper/2403.18697/) (2024)
- [Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation](https://mlsanity.com/paper/2311.15698/) (2023)
- [Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models](https://mlsanity.com/paper/2406.07288/) (2024)
- ["Vorbecsti Romanecste?" A Recipe to Train Powerful Romanian LLMs with English Instructions](https://mlsanity.com/paper/2406.18266/) (2024)
- [Open Llama2 Model for the Lithuanian Language](https://mlsanity.com/paper/2408.12963/) (2024)
- [Unipa-GPT: Large Language Models for university-oriented QA in Italian](https://mlsanity.com/paper/2407.14246/) (2024)

## Abstract (arXiv)

This paper presents Fauno, the first and largest open-source Italian conversational Large Language Model (LLM). Our goal with Fauno is to democratize the study of LLMs in Italian, demonstrating that obtaining a fine-tuned conversational bot with a single GPU is possible. In addition, we release a collection of datasets for conversational AI in Italian. The datasets on which we fine-tuned Fauno include various topics such as general question answering, computer science, and medical questions. We release our code and datasets on urlhttps://github.com/RSTLess-research/Fauno-Italian-LLM