# RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment

arXiv: https://arxiv.org/abs/2604.22520 · HTML: https://mlsanity.com/paper/2604.22520/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects](https://mlsanity.com/paper/2502.12404/) (2025)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations](https://mlsanity.com/paper/2509.12886/) (2025)
- [BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation](https://mlsanity.com/paper/2502.04314/) (2025)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)
- [Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation](https://mlsanity.com/paper/2503.06594/) (2025)
- [Probing the Difficulty Perception Mechanism of Large Language Models](https://mlsanity.com/paper/2510.05969/) (2025)
- [Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral](https://mlsanity.com/paper/2502.12701/) (2025)
- [Combining the Best of Both Worlds: A Method for Hybrid NMT and LLM Translation](https://mlsanity.com/paper/2505.13554/) (2025)

## Built on by (most cited first, 2 of 2)

- [Return or Revise? Learning When Revision Helps Retrieval-Augmented QA](https://mlsanity.com/paper/2609.30087/) (2026)
- [Discovering Translation-Worthy Languages with E-Values](https://mlsanity.com/paper/2609.06593/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT), but deploying them at scale remains prohibitively expensive. A widely adopted remedy is the hybrid system paradigm, which balances cost and quality by serving most requests with a small model and selectively routing a fraction to a large model. However, existing routing strategies often rely on heuristics, external predictors, or absolute quality estimation, which fail to capture whether the large model actually provides a worthwhile improvement over the small one. In this paper, we formulate routing as a budget allocation problem and identify marginal gain, i.e., the large model's improvement over the small model, as the optimal signal for budgeted decisions. Building on this, we propose RouteLMT (routing for LLM-based MT), an efficient in-model router that predicts this expected gain by probing the small translators prompt-token representation, without requiring external models or hypothesis decoding. Extensive experiments demonstrate that our RouteLMT outperforms heuristics, quality/difficulty estimation baselines, achieving a superior quality-budget Pareto frontier. Furthermore, we analyze regression risks and show that a simple guarded variant can mitigate severe quality losses.