# MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction

arXiv: https://arxiv.org/abs/2308.06552 · HTML: https://mlsanity.com/paper/2308.06552/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [Cross-Lingual Natural Language Generation via Pre-Training](https://mlsanity.com/paper/1909.10481/) (2019)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [A Survey on Open Information Extraction](https://mlsanity.com/paper/1806.05599/) (2018)
- [Neural Open Information Extraction](https://mlsanity.com/paper/1805.04270/) (2018)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Multi-task Learning for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.02523/) (2020)
- [CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs](https://mlsanity.com/paper/2112.08804/) (2021)
- [Improving Zero-Shot Translation by Disentangling Positional Information](https://mlsanity.com/paper/2012.15127/) (2020)
- [IMoJIE: Iterative Memory-Based Joint Open Information Extraction](https://mlsanity.com/paper/2005.08178/) (2020)
- [Span Model for Open Information Extraction on Accurate Corpus](https://mlsanity.com/paper/1901.10879/) (2019)
- [Multi2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT](https://mlsanity.com/paper/2009.08128/) (2020)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation](https://mlsanity.com/paper/2210.07022/) (2022)
- [Parameter-Efficient Tuning Makes a Good Classification Head](https://mlsanity.com/paper/2210.16771/) (2022)
- [FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models](https://mlsanity.com/paper/2308.00065/) (2023)
- [ConNER: Consistency Training for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2211.09394/) (2022)
- [UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation](https://mlsanity.com/paper/2207.04900/) (2022)
- [LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction](https://mlsanity.com/paper/2208.10833/) (2022)
- [Open Information Extraction from Question-Answer Pairs](https://mlsanity.com/paper/1903.00172/) (2019)
- [Hybrid Neural Tagging Model for Open Relation Extraction](https://mlsanity.com/paper/1908.01761/) (2019)

## Built on by (most cited first, 3 of 3)

- [OWL: A Large Language Model for IT Operations](https://mlsanity.com/paper/2309.09298/) (2023)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations](https://mlsanity.com/paper/2404.08977/) (2024)

## Abstract (arXiv)

Cross-lingual open information extraction aims to extract structured information from raw text across multiple languages. Previous work uses a shared cross-lingual pre-trained model to handle the different languages but underuses the potential of the language-specific representation. In this paper, we propose an effective multi-stage tuning framework called MT4CrossIE, designed for enhancing cross-lingual open information extraction by injecting language-specific knowledge into the shared model. Specifically, the cross-lingual pre-trained model is first tuned in a shared semantic space (e.g., embedding matrix) in the fixed encoder and then other components are optimized in the second stage. After enough training, we freeze the pre-trained model and tune the multiple extra low-rank language-specific modules using mixture-of-LoRAs for model-based cross-lingual transfer. In addition, we leverage two-stage prompting to encourage the large language model (LLM) to annotate the multi-lingual raw data for data-based cross-lingual transfer. The model is trained with multi-lingual objectives on our proposed dataset OpenIE4++ by combing the model-based and data-based transfer techniques. Experimental results on various benchmarks emphasize the importance of aggregating multiple plug-in-and-play language-specific modules and demonstrate the effectiveness of MT4CrossIE in cross-lingual OIEfootnoteurlhttps://github.com/CSJianYang/Multilingual-Multimodal-NLP.