# RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs

arXiv: https://arxiv.org/abs/2502.07424 · HTML: https://mlsanity.com/paper/2502.07424/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [SelfIE: Self-Interpretation of Large Language Model Embeddings](https://mlsanity.com/paper/2403.10949/) (2024)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization](https://mlsanity.com/paper/2401.14280/) (2024)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models](https://mlsanity.com/paper/2312.10091/) (2023)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)

## Built on by (most cited first, 12 of 12)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders](https://mlsanity.com/paper/2511.10840/) (2025)
- [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025)
- [Multilingual Language Models Encode Script Over Linguistic Structure](https://mlsanity.com/paper/2604.05090/) (2026)
- [Indic-TunedLens: Interpreting Multilingual Models in Indian Languages](https://mlsanity.com/paper/2602.15038/) (2026)
- [Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models](https://mlsanity.com/paper/2605.31220/) (2026)
- [Script Choice in LLMs: Evidence for Late-Layer Commitment](https://mlsanity.com/paper/2609.28784/) (2026)
- [Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration](https://mlsanity.com/paper/2601.02906/) (2026)
- [The Latin Substrate: How Language Models Represent and Mediate Script Choice](https://mlsanity.com/paper/2605.31363/) (2026)
- [Leveraging LaBSE with Progressive Curriculum Learning for Multicultural Polarization](https://mlsanity.com/paper/2606.21718/) (2026)
- [MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2607.24030/) (2026)
- [Why Pretraining Fails to Share Cross-Lingual Knowledge](https://mlsanity.com/paper/2609.19291/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora. This raises a fundamental question: How do LLMs achieve such multilingual capabilities? Focusing on languages written in non-Roman scripts, we investigate the role of Romanization - the representation of non-Roman scripts using Roman characters - as a potential bridge in multilingual processing. Using mechanistic interpretability techniques, we analyze next-token generation and find that intermediate layers frequently represent target words in Romanized form before transitioning to native script, a phenomenon we term Latent Romanization. Further, through activation patching experiments, we demonstrate that LLMs encode semantic concepts similarly across native and Romanized scripts, suggesting a shared underlying representation. Additionally, for translation into non-Roman script languages, our findings reveal that when the target language is in Romanized form, its representations emerge earlier in the model's layers compared to native script. These insights contribute to a deeper understanding of multilingual representation in LLMs and highlight the implicit role of Romanization in facilitating language transfer.