# Improving Translation Faithfulness of Large Language Models via Augmenting Instructions

arXiv: https://arxiv.org/abs/2308.12674 · HTML: https://mlsanity.com/paper/2308.12674/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Domain Robustness in Neural Machine Translation](https://mlsanity.com/paper/1911.03109/) (2019)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [TIM: Teaching Large Language Models to Translate with Comparison](https://mlsanity.com/paper/2307.04408/) (2023)
- [Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer](https://mlsanity.com/paper/2205.11631/) (2022)
- [BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models](https://mlsanity.com/paper/2306.10968/) (2023)
- [Prompt Injection: Parameterization of Fixed Inputs](https://mlsanity.com/paper/2206.11349/) (2022)
- [HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation](https://mlsanity.com/paper/2305.11746/) (2023)
- [ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback](https://mlsanity.com/paper/2304.02426/) (2023)
- [Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners](https://mlsanity.com/paper/2210.02969/) (2022)
- [Is Encoder-Decoder Redundant for Neural Machine Translation?](https://mlsanity.com/paper/2210.11807/) (2022)

## Built on by (most cited first, 28 of 28)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models](https://mlsanity.com/paper/2309.11674/) (2023)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Distract Large Language Models for Automatic Jailbreak Attack](https://mlsanity.com/paper/2403.08424/) (2024)
- [X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale](https://mlsanity.com/paper/2410.03115/) (2024)
- [SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection](https://mlsanity.com/paper/2402.16705/) (2024)
- [G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation](https://mlsanity.com/paper/2405.12915/) (2024)
- [Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models](https://mlsanity.com/paper/2406.15718/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions](https://mlsanity.com/paper/2401.08088/) (2024)
- [A Preference-driven Paradigm for Enhanced Translation with Large Language Models](https://mlsanity.com/paper/2404.11288/) (2024)
- [Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models](https://mlsanity.com/paper/2312.04691/) (2023)
- [On-the-Fly Fusion of Large Language Models and Machine Translation](https://mlsanity.com/paper/2311.08306/) (2023)
- [Context-aware and Style-related Incremental Decoding framework for Discourse-Level Literary Translation](https://mlsanity.com/paper/2409.16539/) (2024)
- [Aligning Translation-Specific Understanding to General Understanding in Large Language Models](https://mlsanity.com/paper/2401.05072/) (2024)
- [Language Modelling Approaches to Adaptive Machine Translation](https://mlsanity.com/paper/2401.14559/) (2024)
- [A Novel Paradigm Boosting Translation Capabilities of Large Language Models](https://mlsanity.com/paper/2403.11430/) (2024)
- [5W1H Extraction With Large Language Models](https://mlsanity.com/paper/2405.16150/) (2024)
- [CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation](https://mlsanity.com/paper/2501.13927/) (2025)
- [POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2401.05596/) (2024)
- [Sing it, Narrate it: Quality Musical Lyrics Translation](https://mlsanity.com/paper/2410.22066/) (2024)
- [Relay Decoding: Concatenating Large Language Models for Machine Translation](https://mlsanity.com/paper/2405.02933/) (2024)
- [Learning Auxiliary Tasks Improves Reference-Free Hallucination Detection in Open-Domain Long-Form Generation](https://mlsanity.com/paper/2505.12265/) (2025)
- [TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection](https://mlsanity.com/paper/2507.03673/) (2025)
- [M-DaQ: Retrieving Samples with Multilingual Diversity and Quality for Instruction Fine-Tuning Datasets](https://mlsanity.com/paper/2509.15549/) (2025)
- [Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention](https://mlsanity.com/paper/2608.26168/) (2026)
- [TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs](https://mlsanity.com/paper/2609.11399/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) present strong general capabilities, and a current compelling challenge is stimulating their specialized capabilities, such as machine translation, through low-cost instruction tuning. The standard instruction-following data is sequentially organized as the concatenation of an instruction, an input, and a response. As the attention mechanism of LLMs has limitations on local focus, LLMs tend to focus more on the words or sentences nearby at each position. This leads to a high risk of instruction forgetting during decoding. To alleviate the above issues, We propose SWIE (Segment-Weighted Instruction Embedding) and an instruction-following dataset OVERMISS. SWIE improves the model instruction understanding by adding a global instruction representation on the following input and response representations. OVERMISS improves model faithfulness by comparing over-translation and miss-translation results with the correct translation. We apply our methods to two main-stream open-source LLMs, BLOOM and LLaMA. The experimental results demonstrate significant improvements in translation performance with SWIE based on BLOOMZ-3b, particularly in zero-shot and long text translations due to reduced instruction forgetting risk. Additionally, OVERMISS outperforms the baseline in translation performance (e.g. an increase in BLEU scores from 0.69 to 3.12 and an average improvement of 0.48 percentage comet scores for LLaMA-7b) with further enhancements seen in models combining OVERMISS and SWIE (e.g. the BLUE scores increase up to 0.56 from English to German across three different backbones), and both exhibit improvements in the faithfulness metric based on word alignment.