# Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing

arXiv: https://arxiv.org/abs/2410.07054 · HTML: https://mlsanity.com/paper/2410.07054/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions](https://mlsanity.com/paper/2202.11912/) (2022)
- [Visualizing Deep Networks by Optimizing with Integrated Gradients](https://mlsanity.com/paper/1905.00954/) (2019)
- [Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://mlsanity.com/paper/2303.01911/) (2023)
- [Adaptive Machine Translation with Large Language Models](https://mlsanity.com/paper/2301.13294/) (2023)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)

## Built on by (most cited first, 18 of 18)

- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2503.23064/) (2025)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [Repetitions are not all alike: distinct mechanisms sustain repetition in language models](https://mlsanity.com/paper/2504.01100/) (2025)
- [What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers](https://mlsanity.com/paper/2506.13688/) (2025)
- [CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks](https://mlsanity.com/paper/2505.07261/) (2025)
- [Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective](https://mlsanity.com/paper/2601.20520/) (2026)
- [MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages](https://mlsanity.com/paper/2503.01150/) (2025)
- [Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning](https://mlsanity.com/paper/2506.18330/) (2025)
- [Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning](https://mlsanity.com/paper/2507.07810/) (2025)
- [Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval](https://mlsanity.com/paper/2507.23284/) (2025)
- [OmniGen-AR: AutoRegressive Any-to-Image Generation](https://mlsanity.com/paper/2606.09156/) (2026)
- [Quality-Aware Translation Tagging in Multilingual RAG system](https://mlsanity.com/paper/2510.23070/) (2025)
- [MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation](https://mlsanity.com/paper/2604.04839/) (2026)
- [Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors](https://mlsanity.com/paper/2502.19669/) (2025)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://mlsanity.com/paper/2601.13729/) (2026)
- [DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines](https://mlsanity.com/paper/2604.14314/) (2026)
- [TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs](https://mlsanity.com/paper/2609.11399/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have recently revolutionized the NLP field, while they still fall short in some specific down-stream tasks. In the work, we focus on utilizing LLMs to perform machine translation, where we observe that two patterns of errors frequently occur and drastically affect the translation quality: language mismatch and repetition. The work sets out to explore the potential for mitigating these two issues by leveraging model editing methods, e.g., by locating Feed-Forward Network (FFN) neurons or something that are responsible for the errors and deactivating them in the inference time. We find that directly applying such methods either limited effect on the targeted errors or has significant negative side-effect on the general translation quality, indicating that the located components may also be crucial for ensuring machine translation with LLMs on the rails. To this end, we propose to refine the located components by fetching the intersection of the locating results under different language settings, filtering out the aforementioned information that is irrelevant to targeted errors. The experiment results empirically demonstrate that our methods can effectively reduce the language mismatch and repetition ratios and meanwhile enhance or keep the general translation quality in most cases.