# Cross-Prompt Encoder for Low-Performing Languages

arXiv: https://arxiv.org/abs/2508.10352 · HTML: https://mlsanity.com/paper/2508.10352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.02861/) (2023)
- [Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization](https://mlsanity.com/paper/2305.03937/) (2023)
- [Consistency Regularization for Cross-Lingual Fine-Tuning](https://mlsanity.com/paper/2106.08226/) (2021)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)
- [Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer](https://mlsanity.com/paper/2305.12761/) (2023)
- [Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More](https://mlsanity.com/paper/2402.03782/) (2024)
- [Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator](https://mlsanity.com/paper/2403.12407/) (2024)
- [Language Fusion for Parameter-Efficient Cross-lingual Transfer](https://mlsanity.com/paper/2501.06892/) (2025)
- [Enhancing Small Language Models for Cross-Lingual Generalized Zero-Shot Classification with Soft Prompt Tuning](https://mlsanity.com/paper/2503.19469/) (2025)
- [DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2505.15090/) (2025)

## Abstract (arXiv)

Soft prompts have emerged as a powerful alternative to adapters in parameter-efficient fine-tuning (PEFT), enabling large language models (LLMs) to adapt to downstream tasks without architectural changes or parameter updates. While prior work has focused on stabilizing training via parameter interaction in small neural prompt encoders, their broader potential for transfer across languages remains unexplored. In this paper, we demonstrate that a prompt encoder can play a central role in improving performance on low-performing languages - those that achieve poor accuracy even under full-model fine-tuning. We investigate a lightweight encoder paired with multi-source training on typologically diverse languages. We call this architecture-training combination the Cross-Prompt Encoder (XPE), and show that it advances the capture of abstract, transferable patterns across languages. To complement XPE, we propose a Dual Soft Prompt mechanism that combines an encoder-based prompt with a directly trained standard soft prompt. This hybrid design proves especially effective for target languages that benefit from both broadly shared structure and language-specific alignment. Text classification experiments with a transformer encoder (XLM-R) on the SIB-200 benchmark reveal a consistent trade-off: XPE is most effective for low-performing languages, while hybrid variants offer broader adaptability across multilingual settings.