# Data Annealing for Informal Language Understanding Tasks

arXiv: https://arxiv.org/abs/2004.13833 · HTML: https://mlsanity.com/paper/2004.13833/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 4 of 4)

- [Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions](https://mlsanity.com/paper/2311.17048/) (2023)
- [Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units](https://mlsanity.com/paper/2106.04935/) (2021)
- [Gender Recognition in Informal and Formal Language Scenarios via Transfer Learning](https://mlsanity.com/paper/2107.02759/) (2021)
- [PREMISE: Matching-based Prediction for Accurate Review Recommendation](https://mlsanity.com/paper/2505.01255/) (2025)

## Abstract (arXiv)

There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We pro-pose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.