# Why Not Transform Chat Large Language Models to Non-English?

arXiv: https://arxiv.org/abs/2405.13923 · HTML: https://mlsanity.com/paper/2405.13923/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization](https://mlsanity.com/paper/2401.06838/) (2024)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning](https://mlsanity.com/paper/2311.08711/) (2023)
- [LLM Augmented LLMs: Expanding Capabilities through Composition](https://mlsanity.com/paper/2401.02412/) (2024)
- [Low-Resource Corpus Filtering using Multilingual Sentence Embeddings](https://mlsanity.com/paper/1906.08885/) (2019)

## Built on by (most cited first, 5 of 5)

- [Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering](https://mlsanity.com/paper/2511.23231/) (2025)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [Trans-Zero: Self-Play Incentivizes Large Language Models for Multilingual Translation Without Parallel Data](https://mlsanity.com/paper/2504.14669/) (2025)
- [Text2Cypher Across Languages: Evaluating and Finetuning LLMs](https://mlsanity.com/paper/2506.21445/) (2025)
- [Incremental Multilingual Text2Cypher with Adapter Combination](https://mlsanity.com/paper/2601.16097/) (2026)

## Abstract (arXiv)

The scarcity of non-English data limits the development of non-English large language models (LLMs). Transforming English-centric LLMs to non-English has been identified as an effective and resource-efficient method. Previous works start from base LLMs and perform knowledge distillation (KD) with data generated by stronger LLMs, e.g. GPT-4. Compared to base LLMs, chat LLMs are further optimized for advanced abilities, e.g. multi-turn conversation and human preference alignment, and thus more powerful in both helpfulness and safety. However, transforming a chat LLM involves two critical issues: (1) How can we effectively transfer advanced abilities without their supervised data? (2) How can we prevent the original knowledge from catastrophic forgetting during transformation? We target these issues by introducing a simple framework called TransLLM. For the first issue, TransLLM divides the transfer problem into some common sub-tasks with the translation chain-of-thought, which uses the translation as the bridge between English and non-English step-by-step. We further enhance the performance of sub-tasks with publicly available data. For the second issue, we propose a method comprising two synergistic components: low-rank adaptation for training to maintain the original LLM parameters, and recovery KD, which utilizes data generated by the chat LLM itself to recover the original knowledge from the frozen parameters. In the experiments, we transform the LLaMA-2-chat-7B to the Thai language. Our method, using only single-turn data, outperforms strong baselines and ChatGPT on multi-turn benchmark MT-bench. Furthermore, our method, without safety data, rejects more harmful queries of safety benchmark AdvBench than both ChatGPT and GPT-4. Code is available at https://github.com/hy5468/TransLLM.