# TIER-A: Denoising Learning Framework for Information Extraction

arXiv: https://arxiv.org/abs/2211.11527 · HTML: https://mlsanity.com/paper/2211.11527/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 13 Nov 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution](https://mlsanity.com/paper/1909.12978/) (2019)
- [Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning](https://mlsanity.com/paper/1805.09927/) (2018)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)

## Abstract (arXiv)

With the development of deep neural language models, great progress has been made in information extraction recently. However, deep learning models often overfit on noisy data points, leading to poor performance. In this work, we examine the role of information entropy in the overfitting process and draw a key insight that overfitting is a process of overconfidence and entropy decreasing. Motivated by such properties, we propose a simple yet effective co-regularization joint-training framework TIER-A, Aggregation Joint-training Framework with Temperature Calibration and Information Entropy Regularization. Our framework consists of several neural models with identical structures. These models are jointly trained and we avoid overfitting by introducing temperature and information entropy regularization. Extensive experiments on two widely-used but noisy datasets, TACRED and CoNLL03, demonstrate the correctness of our assumption and the effectiveness of our framework.