# How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes

arXiv: https://arxiv.org/abs/2409.03454 · HTML: https://mlsanity.com/paper/2409.03454/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Neural Machine Translation with Monolingual Translation Memory](https://mlsanity.com/paper/2105.11269/) (2021)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [Adaptive Machine Translation with Large Language Models](https://mlsanity.com/paper/2301.13294/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning](https://mlsanity.com/paper/2310.13448/) (2023)
- [Fine-tuning Large Language Models for Domain-specific Machine Translation](https://mlsanity.com/paper/2402.15061/) (2024)
- [Regularization techniques for fine-tuning in neural machine translation](https://mlsanity.com/paper/1707.09920/) (2017)
- [ChatGPT MT: Competitive for High- (but not Low-) Resource Languages](https://mlsanity.com/paper/2309.07423/) (2023)
- [Augmenting Large Language Model Translators via Translation Memories](https://mlsanity.com/paper/2305.17367/) (2023)
- [Fine-tuning Large Language Models for Adaptive Machine Translation](https://mlsanity.com/paper/2312.12740/) (2023)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [Domain-Specific Text Generation for Machine Translation](https://mlsanity.com/paper/2208.05909/) (2022)
- [Translation Memory Retrieval Methods](https://mlsanity.com/paper/1505.05841/) (2015)

## Built on by (most cited first, 17 of 17)

- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [On the generalization of language models from in-context learning and finetuning: a controlled study](https://mlsanity.com/paper/2505.00661/) (2025)
- [Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation](https://mlsanity.com/paper/2510.19897/) (2025)
- [LlamaRestTest: Effective REST API Testing with Small Language Models](https://mlsanity.com/paper/2501.08598/) (2025)
- [Translation Analytics for Freelancers: I. Introduction, Data Preparation, Baseline Evaluations](https://mlsanity.com/paper/2504.14619/) (2025)
- [Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs](https://mlsanity.com/paper/2505.22293/) (2025)
- [Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection](https://mlsanity.com/paper/2506.03476/) (2025)
- [Translating Under Pressure: Domain-Aware LLMs for Crisis Communication](https://mlsanity.com/paper/2604.26597/) (2026)
- [MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning](https://mlsanity.com/paper/2607.16789/) (2026)
- [Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study](https://mlsanity.com/paper/2502.16022/) (2025)
- [Look the Other Way: Designing 'Positive' Molecules with Negative Data via Task Arithmetic](https://mlsanity.com/paper/2507.17876/) (2025)
- [Real-Time RAG for the Identification of Supply Chain Vulnerabilities](https://mlsanity.com/paper/2509.10469/) (2025)
- [Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation](https://mlsanity.com/paper/2512.10734/) (2025)
- [Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness](https://mlsanity.com/paper/2604.01601/) (2026)
- [An Empirical Study of Many-Shot In-Context Learning for Machine Translation of Low-Resource Languages](https://mlsanity.com/paper/2604.02596/) (2026)
- [Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysis](https://mlsanity.com/paper/2605.06330/) (2026)
- [Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows](https://mlsanity.com/paper/2605.31452/) (2026)

## Abstract (arXiv)

Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. However, LLMs often struggle with the nuances and style required for organisation-specific translation. In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. Our findings reveal improvement in translation performance with larger datasets across all metrics. On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains.