# Towards Linguistically Informed Multi-Objective Pre-Training for Natural Language Inference

arXiv: https://arxiv.org/abs/2212.07428 · HTML: https://mlsanity.com/paper/2212.07428/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [Self-Explaining Structures Improve NLP Models](https://mlsanity.com/paper/2012.01786/) (2020)
- [LIMIT-BERT : Linguistic Informed Multi-Task BERT](https://mlsanity.com/paper/1910.14296/) (2019)
- [Informed Pre-Training on Prior Knowledge](https://mlsanity.com/paper/2205.11433/) (2022)
- [Contradiction Detection in Persian Text](https://mlsanity.com/paper/2107.01987/) (2021)

## Built on by (most cited first, 1 of 1)

- [Generating Prototypes for Contradiction Detection Using Large Language Models and Linguistic Rules](https://mlsanity.com/paper/2310.14732/) (2023)

## Abstract (arXiv)

We introduce a linguistically enhanced combination of pre-training methods for transformers. The pre-training objectives include POS-tagging, synset prediction based on semantic knowledge graphs, and parent prediction based on dependency parse trees. Our approach achieves competitive results on the Natural Language Inference task, compared to the state of the art. Specifically for smaller models, the method results in a significant performance boost, emphasizing the fact that intelligent pre-training can make up for fewer parameters and help building more efficient models. Combining POS-tagging and synset prediction yields the overall best results.