# Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs

arXiv: https://arxiv.org/abs/2506.17080 · HTML: https://mlsanity.com/paper/2506.17080/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [HelpSteer2: Open-source dataset for training top-performing reward models](https://mlsanity.com/paper/2406.08673/) (2024)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects](https://mlsanity.com/paper/2502.12404/) (2025)
- [WPO: Enhancing RLHF with Weighted Preference Optimization](https://mlsanity.com/paper/2406.11827/) (2024)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale](https://mlsanity.com/paper/2410.03115/) (2024)
- [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://mlsanity.com/paper/2504.01001/) (2025)
- [Recovering document annotations for sentence-level bitext](https://mlsanity.com/paper/2406.03869/) (2024)
- [Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation](https://mlsanity.com/paper/2503.08327/) (2025)
- [Post-edits Are Preferences Too](https://mlsanity.com/paper/2410.02320/) (2024)

## Built on by (most cited first, 30 of 40)

- [MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task](https://mlsanity.com/paper/2510.24707/) (2025)
- [EuroLLM-22B: Technical Report](https://mlsanity.com/paper/2602.05879/) (2026)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Making, not Taking, the Best of N](https://mlsanity.com/paper/2510.00931/) (2025)
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://mlsanity.com/paper/2601.07338/) (2026)
- [Test-Time Scaling of Reasoning Models for Machine Translation](https://mlsanity.com/paper/2510.06471/) (2025)
- [Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model](https://mlsanity.com/paper/2509.25543/) (2025)
- [M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation](https://mlsanity.com/paper/2510.13434/) (2025)
- [Scaling Model and Data for Multilingual Machine Translation with Open Large Language Models](https://mlsanity.com/paper/2602.11961/) (2026)
- [Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation](https://mlsanity.com/paper/2603.13045/) (2026)
- [From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task](https://mlsanity.com/paper/2508.12774/) (2025)
- [Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework](https://mlsanity.com/paper/2602.14073/) (2026)
- [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025)
- [LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning](https://mlsanity.com/paper/2510.09189/) (2025)
- [TowerVision: Understanding and Improving Multilinguality in Vision-Language Models](https://mlsanity.com/paper/2510.21849/) (2025)
- [Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models](https://mlsanity.com/paper/2606.08451/) (2026)
- [Augmenting Text to Increase Translation Difficulty](https://mlsanity.com/paper/2608.15932/) (2026)
- [STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation](https://mlsanity.com/paper/2608.27161/) (2026)
- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)
- [Prompt Revision as a Source of Cultural Bias in Text-to-Image Systems](https://mlsanity.com/paper/2609.11532/) (2026)
- [Learning to Translate Ambiguous Terminology by Preference Optimization on Post-Edits](https://mlsanity.com/paper/2507.03580/) (2025)
- [Self-Speculative Biased Decoding for Faster Re-Translation](https://mlsanity.com/paper/2509.21740/) (2025)
- [Stronger Re-identification Attacks through Reasoning and Aggregation](https://mlsanity.com/paper/2510.09184/) (2025)
- [Can QE-informed (Re)Translation lead to Error Correction?](https://mlsanity.com/paper/2511.13884/) (2025)
- [Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography](https://mlsanity.com/paper/2602.14675/) (2026)
- [Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling](https://mlsanity.com/paper/2604.28075/) (2026)
- [Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation](https://mlsanity.com/paper/2605.07533/) (2026)
- [Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows](https://mlsanity.com/paper/2605.31452/) (2026)
- [Worlds Within Words: Translating Culture in Ancient Chinese Texts with Multi-Agent Coordination](https://mlsanity.com/paper/2606.01276/) (2026)
- [Automatic Labelling of Speech Translation Errors](https://mlsanity.com/paper/2606.06047/) (2026)

## Abstract (arXiv)

Fine-tuning pretrained LLMs has been shown to be an effective strategy for reaching state-of-the-art performance on specific tasks like machine translation. However, this process of adaptation often implies sacrificing general-purpose capabilities, such as conversational reasoning and instruction-following, hampering the utility of the system in real-world applications that require a mixture of skills. In this paper, we introduce Tower+, a suite of models designed to deliver strong performance across both translation and multilingual general-purpose text capabilities. We achieve a Pareto frontier between translation specialization and multilingual general-purpose capabilities by introducing a novel training recipe that builds on Tower (Alves et al., 2024), comprising continued pretraining, supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards. At each stage of training, we carefully generate and curate data to strengthen performance on translation as well as general-purpose tasks involving code generation, mathematics problem solving, and general instruction-following. We develop models at multiple scales: 2B, 9B, and 72B. Our smaller models often outperform larger general-purpose open-weight and proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers best-in-class translation performance for high-resource languages and top results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we introduce for evaluating both translation and instruction-following. Our findings highlight that it is possible to rival frontier models in general capabilities, while optimizing for specific business domains, such as translation and localization.