# A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations

arXiv: https://arxiv.org/abs/2109.04727 · HTML: https://mlsanity.com/paper/2109.04727/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Rethinking embedding coupling in pre-trained language models](https://mlsanity.com/paper/2010.12821/) (2020)
- [How Language-Neutral is Multilingual BERT?](https://mlsanity.com/paper/1911.03310/) (2019)
- [Explicit Alignment Objectives for Multilingual Bidirectional Encoders](https://mlsanity.com/paper/2010.07972/) (2020)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model](https://mlsanity.com/paper/1810.12836/) (2018)
- [LAReQA: Language-agnostic answer retrieval from a multilingual pool](https://mlsanity.com/paper/2004.05484/) (2020)
- [Universal Sentence Representation Learning with Conditional Masked Language Model](https://mlsanity.com/paper/2012.14388/) (2020)
- [ReQA: An Evaluation for End-to-End Answer Retrieval Models](https://mlsanity.com/paper/1907.04780/) (2019)
- [Parameter-free Sentence Embedding via Orthogonal Basis](https://mlsanity.com/paper/1810.00438/) (2018)

## Built on by (most cited first, 17 of 17)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Representational Analysis of Binding in Language Models](https://mlsanity.com/paper/2409.05448/) (2024)
- [Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations](https://mlsanity.com/paper/2401.05792/) (2024)
- [How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning](https://mlsanity.com/paper/2305.13286/) (2023)
- [Automatic Rule Induction for Interpretable Semi-Supervised Learning](https://mlsanity.com/paper/2205.09067/) (2022)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Language Agnostic Code Embeddings](https://mlsanity.com/paper/2310.16803/) (2023)
- [Linear Cross-Lingual Mapping of Sentence Embeddings](https://mlsanity.com/paper/2305.14256/) (2023)
- [Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment](https://mlsanity.com/paper/2407.14878/) (2024)
- [Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features](https://mlsanity.com/paper/2408.01394/) (2024)
- [LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal](https://mlsanity.com/paper/2601.04768/) (2026)
- [Language steering in latent space to mitigate unintended code-switching](https://mlsanity.com/paper/2510.13849/) (2025)
- [SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval](https://mlsanity.com/paper/2606.18801/) (2026)
- [Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing](https://mlsanity.com/paper/2410.00513/) (2024)
- [TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B](https://mlsanity.com/paper/2510.06249/) (2025)
- [Retrieval Disagreement as a Signal: Diagnosing and Steering Multilingual Dense Embedders](https://mlsanity.com/paper/2610.18526/) (2026)

## Abstract (arXiv)

Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. We explore this problem from a novel angle of geometric algebra and semantic space. A simple but highly effective method "Language Information Removal (LIR)" factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data. A post-training and model-agnostic method, LIR only uses simple linear operations, e.g. matrix factorization and orthogonal projection. LIR reveals that for weak-alignment multilingual systems, the principal components of semantic spaces primarily encodes language identity information. We first evaluate the LIR on a cross-lingual question answer retrieval task (LAReQA), which requires the strong alignment for the multilingual embedding space. Experiment shows that LIR is highly effectively on this task, yielding almost 100% relative improvement in MAP for weak-alignment models. We then evaluate the LIR on Amazon Reviews and XEVAL dataset, with the observation that removing language information is able to improve the cross-lingual transfer performance.