mlsanityData to 13 Oct 2026

Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languages

arXiv 2411.02398 ↗First posted 4 Nov 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Although multilingual LLMs have achieved remarkable performance across benchmarks, we find they continue to underperform on non-Latin script languages across contemporary LLM families. This discrepancy arises from the fact that LLMs are pretrained with orthographic scripts, which are dominated by Latin characters that obscure their shared phonology with non-Latin scripts. We propose leveraging phonemic transcriptions as complementary signals to induce script-invariant representations. Our study demonstrates that integrating phonemic signals improves performance across both non-Latin and Latin script languages, with a particularly significant impact on closing the performance gap between the two. Through detailed experiments, we show that phonemic and orthographic scripts retrieve distinct examples for in-context learning (ICL). This motivates our proposed Mixed-ICL retrieval strategy, where further aggregation from both leads to our significant performance improvements for both Latin script languages (up to 12.6%) and non-Latin script languages (up to 15.1%) compared to randomized ICL retrieval.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 44 of its 82 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

44 of 82 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2024The Llama 3 Herd of Modelscited by 14,486
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  11. 2024Qwen2 Technical Reportcited by 2,102
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,302
32 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,288
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,126
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  6. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 263
  8. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  9. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 205
  10. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  13. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  14. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  15. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  16. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  17. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  18. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  19. 2018Learning To Split and Rephrase From Wikipedia Edit Historycited by 38
  20. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  21. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  22. 2019Multi-Level Matching and Aggregation Network for Few-Shot Relation Classificationcited by 32
  23. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  24. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  25. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  26. 2020CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detectioncited by 23
  27. 2023CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Taskscited by 18
  28. 2020Dynamic Semantic Matching and Aggregation Network for Few-shot Intent Detectioncited by 17
  29. 2023Enhancing Cross-lingual Transfer via Phonemic Transcription Integrationcited by 11
  30. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  31. 2023Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learningcited by 4
  32. 2024CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scriptscited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Happiness is Sharing a Vocabulary: A Study of Transliteration Methodscited by 2
  2. 2025BIRD: Bronze Inscription Restoration and Datingcited by 2
  3. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  4. 2025What Really Counts? Examining Step and Token Level Attribution in Multilingual CoT Reasoningcited by 1
  5. 2026Leveraging LaBSE with Progressive Curriculum Learning for Multicultural Polarizationcited by 0
  6. 2026CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settingscited by 0