# Multilingual Constituency Parsing with Self-Attention and Pre-Training

arXiv: https://arxiv.org/abs/1812.11760 · HTML: https://mlsanity.com/paper/1812.11760/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2018.

## Built on by (most cited first, 30 of 117)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models](https://mlsanity.com/paper/2307.14430/) (2023)
- [The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning](https://mlsanity.com/paper/2305.14045/) (2023)
- [Data Diversity Matters for Robust Instruction Tuning](https://mlsanity.com/paper/2311.14736/) (2023)
- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2409.05840/) (2024)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2401.15269/) (2024)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [Bottom-Up Constituency Parsing and Nested Named Entity Recognition with Pointer Networks](https://mlsanity.com/paper/2110.05419/) (2021)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Fine-tuning Large Language Models with Sequential Instructions](https://mlsanity.com/paper/2403.07794/) (2024)
- [LFTK: Handcrafted Features in Computational Linguistics](https://mlsanity.com/paper/2305.15878/) (2023)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Subword Pooling Makes a Difference](https://mlsanity.com/paper/2102.10864/) (2021)
- [X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions](https://mlsanity.com/paper/2405.19744/) (2024)
- [LLMs syntactically adapt their language use to their conversational partner](https://mlsanity.com/paper/2503.07457/) (2025)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation](https://mlsanity.com/paper/2402.10646/) (2024)
- [Predicting Attention Sparsity in Transformers](https://mlsanity.com/paper/2109.12188/) (2021)
- [A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models](https://mlsanity.com/paper/2201.05601/) (2022)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [AutoTSG: Learning and Synthesis for Incident Troubleshooting](https://mlsanity.com/paper/2205.13457/) (2022)
- [Modeling structure-building in the brain with CCG parsing and large language models](https://mlsanity.com/paper/2210.16147/) (2022)
- [World Models for Math Story Problems](https://mlsanity.com/paper/2306.04347/) (2023)
- [FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only](https://mlsanity.com/paper/2408.01323/) (2024)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)

## Abstract (arXiv)

We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText, ELMo, and BERT for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings. We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages. Finally, we demonstrate new state-of-the-art results for 11 languages, including English (95.8 F1) and Chinese (91.8 F1).