# LIMIT-BERT : Linguistic Informed Multi-Task BERT

arXiv: https://arxiv.org/abs/1910.14296 · HTML: https://mlsanity.com/paper/1910.14296/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2019.

## Built on by (most cited first, 10 of 10)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Dialogue-oriented Pre-training](https://mlsanity.com/paper/2106.00420/) (2021)
- [Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning](https://mlsanity.com/paper/2108.00356/) (2021)
- [Word Boundary Information Isn't Useful for Encoder Language Models](https://mlsanity.com/paper/2401.07923/) (2024)
- [Span Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2108.12848/) (2021)
- [A Linguistic Investigation of Machine Learning based Contradiction Detection Models: An Empirical Analysis and Future Perspectives](https://mlsanity.com/paper/2210.10434/) (2022)
- [Towards Linguistically Informed Multi-Objective Pre-Training for Natural Language Inference](https://mlsanity.com/paper/2212.07428/) (2022)
- [UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models](https://mlsanity.com/paper/2305.01624/) (2023)
- [Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?](https://mlsanity.com/paper/2311.00268/) (2023)

## Abstract (arXiv)

In this paper, we present a Linguistic Informed Multi-Task BERT (LIMIT-BERT) for learning language representations across multiple linguistic tasks by Multi-Task Learning (MTL). LIMIT-BERT includes five key linguistic syntax and semantics tasks: Part-Of-Speech (POS) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (SRL). Besides, LIMIT-BERT adopts linguistics mask strategy: Syntactic and Semantic Phrase Masking which mask all of the tokens corresponding to a syntactic/semantic phrase. Different from recent Multi-Task Deep Neural Networks (MT-DNN) (Liu et al., 2019), our LIMIT-BERT is linguistically motivated and learning in a semi-supervised method which provides large amounts of linguistic-task data as same as BERT learning corpus. As a result, LIMIT-BERT not only improves linguistic tasks performance but also benefits from a regularization effect and linguistic information that leads to more general representations to help adapt to new tasks and domains. LIMIT-BERT obtains new state-of-the-art or competitive results on both span and dependency semantic parsing on Propbank benchmarks and both dependency and constituent syntactic parsing on Penn Treebank.