# FootGPT : A Large Language Model Development Experiment on a Minimal Setting

arXiv: https://arxiv.org/abs/2308.08610 · HTML: https://mlsanity.com/paper/2308.08610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Aligning Large Language Models through Synthetic Feedback](https://mlsanity.com/paper/2305.13735/) (2023)
- [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study](https://mlsanity.com/paper/2307.08072/) (2023)
- [INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation](https://mlsanity.com/paper/2306.08162/) (2023)
- [Learning Multi-Step Reasoning by Solving Arithmetic Tasks](https://mlsanity.com/paper/2306.01707/) (2023)
- [A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directions](https://mlsanity.com/paper/2212.13465/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)

## Abstract (arXiv)

With recent empirical observations, it has been argued that the most significant aspect of developing accurate language models may be the proper dataset content and training strategy compared to the number of neural parameters, training duration or dataset size. Following this argument, we opted to fine tune a one billion parameter size trained general purpose causal language model with a dataset curated on team statistics of the Italian football league first ten game weeks, using low rank adaptation. The limited training dataset was compiled based on a framework where a powerful commercial large language model provides distilled paragraphs and question answer pairs as intended. The training duration was kept relatively short to provide a basis for our minimal setting exploration. We share our key observations on the process related to developing a specific purpose language model which is intended to interpret soccer data with constrained resources in this article.