# Temporal-aware Language Representation Learning From Crowdsourced Labels

arXiv: https://arxiv.org/abs/2107.07958 · HTML: https://mlsanity.com/paper/2107.07958/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks](https://mlsanity.com/paper/1706.00038/) (2017)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Triplet-Center Loss for Multi-View 3D Object Retrieval](https://mlsanity.com/paper/1803.06189/) (2018)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations](https://mlsanity.com/paper/1902.08918/) (2019)
- [Representation Learning from Limited Educational Data with Crowdsourced Labels](https://mlsanity.com/paper/2009.11222/) (2020)
- [Learning Effective Embeddings From Crowdsourced Labels: An Educational Case Study](https://mlsanity.com/paper/1908.00086/) (2019)

## Abstract (arXiv)

Learning effective language representations from crowdsourced labels is crucial for many real-world machine learning tasks. A challenging aspect of this problem is that the quality of crowdsourced labels suffer high intra- and inter-observer variability. Since the high-capacity deep neural networks can easily memorize all disagreements among crowdsourced labels, directly applying existing supervised language representation learning algorithms may yield suboptimal solutions. In this paper, we propose TACMA, a underlinetemporal-underlineaware language representation learning heuristic for underlinecrowdsourced labels with underlinemultiple underlineannotators. The proposed approach (1) explicitly models the intra-observer variability with attention mechanism; (2) computes and aggregates per-sample confidence scores from multiple workers to address the inter-observer disagreements. The proposed heuristic is extremely easy to implement in around 5 lines of code. The proposed heuristic is evaluated on four synthetic and four real-world data sets. The results show that our approach outperforms a wide range of state-of-the-art baselines in terms of prediction accuracy and AUC. To encourage the reproducible results, we make our code publicly available at urlhttps://github.com/CrowdsourcingMining/TACMA.