# Prix-LM: Pretraining for Multilingual Knowledge Base Construction arXiv: https://arxiv.org/abs/2110.08443 ยท HTML: https://mlsanity.com/paper/2110.08443/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 69 references that are in the map. - First posted on arXiv 16 Oct 2021. ## Builds on (most cited first, 30 of 39) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019) - [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020) - [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020) - [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016) - [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019) - [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017) - [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017) - [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019) - [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019) - [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020) - [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019) - [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019) - [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020) - [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018) - [DKN: Deep Knowledge-Aware Network for News Recommendation](https://mlsanity.com/paper/1801.08284/) (2018) - [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021) - [Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment](https://mlsanity.com/paper/1611.03954/) (2016) - [A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs](https://mlsanity.com/paper/2003.07743/) (2020) - [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020) - [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016) - [Knowledge Graph Alignment Network with Gated Multi-hop Neighborhood Aggregation](https://mlsanity.com/paper/1911.08936/) (2019) - [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020) - [Learning Intents behind Interactions with Knowledge Graph for Recommendation](https://mlsanity.com/paper/2102.07057/) (2021) - [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021) ## Built on by (most cited first, 12 of 12) - [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023) - [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023) - [GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models](https://mlsanity.com/paper/2205.12247/) (2022) - [SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation](https://mlsanity.com/paper/2210.12213/) (2022) - [Localized Cultural Knowledge is Conserved and Controllable in Large Language Models](https://mlsanity.com/paper/2504.10191/) (2025) - [Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples](https://mlsanity.com/paper/2111.10962/) (2021) - [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022) - [Can Pretrained Language Models (Yet) Reason Deductively?](https://mlsanity.com/paper/2210.06442/) (2022) - [Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints](https://mlsanity.com/paper/2406.18085/) (2024) - [Multilingual Knowledge Graph Completion via Efficient Multilingual Knowledge Sharing](https://mlsanity.com/paper/2510.07736/) (2025) - [Multilingual Information Retrieval with a Monolingual Knowledge Base](https://mlsanity.com/paper/2506.02527/) (2025) - [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024) ## Abstract (arXiv) Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource- and time-intensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones. To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified representation model, Prix-LM, for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines.