TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B
Not on the map. Why?
Abstract (arXiv)
The 2025 Multimodal Models for Low-Resource Contexts and Social Impact (MMLoSo) Language Challenge addresses one of India's most pressing linguistic gaps: the lack of resources for its diverse low-resource languages (LRLs). In this study, we investigate whether enforcing cross-lingual similarity in specific internal layers of a decoder-only multilingual large language model (LLM) can improve translation quality from LRL to high-resource language (HRL). Specifically, we combine Centered Kernel Alignment (CKA), a similarity metric that encourages representations of different languages to align, with REPINA, a regularization method that constrains parameter updates to remain close to the pretrained model, into a joint method we call TRepLiNa. In this research project, we experiment with zero-shot, few-shot, and fine-tuning settings using Aya-23 8B with QLoRA across MMLoSo shared task language pairs (Mundari, Santali, Bhili) with Hindi/English pivots. Our results show that aligning mid-level layers using TRepLiNa (CKA+REPINA) is a low-cost, practical approach to improving LRL translation, especially in data-scarce settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 3 Oct 2025.
Builds on
16 of 33 references.
References · 16
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
- 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
- 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
- 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
- 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
- 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
- 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
- 2022Pre-trained Language Models Can be Fully Zero-Shot Learnerscited by 22
- 2021A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representationscited by 17
- 2022Representation Projection Invariance Mitigates Representation Collapsecited by 10
4 more
- 2024PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Bankscited by 5
- 2025What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representationscited by 5
- 2024Towards Santali Linguistic Inclusion: Building the First Santali-to-English Translation Model using mT5 Transformer and Data Augmentationcited by 1
- 2025Wave-Based Semantic Memory with Resonance-Based Retrieval: A Phase-Aware Alternative to Vector Embedding Storescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study
- 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- 2025Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning
- 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages
- 2025Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus
- 2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders
- 2025A Survey on Training-free Alignment of Large Language Models
- 2026BM25-Augmented Many-Shot Translation for Low-Resource North-Eastern Indian Languages