# Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models

arXiv: https://arxiv.org/abs/2503.21000 · HTML: https://mlsanity.com/paper/2503.21000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 71 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022)
- [A Survey on Ensemble Learning under the Era of Deep Learning](https://mlsanity.com/paper/2101.08387/) (2021)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Multi-label Categorization of Accounts of Sexism using a Neural Framework](https://mlsanity.com/paper/1910.04602/) (2019)
- [Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game](https://mlsanity.com/paper/1506.04744/) (2015)
- [Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests](https://mlsanity.com/paper/2101.06351/) (2021)
- [Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases](https://mlsanity.com/paper/1909.12911/) (2019)
- [BotBuster: Multi-platform Bot Detection Using A Mixture of Experts](https://mlsanity.com/paper/2207.13658/) (2022)
- [Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method](https://mlsanity.com/paper/2305.20042/) (2023)
- [It Takes Two to Negotiate: Modeling Social Exchange in Online Multiplayer Games](https://mlsanity.com/paper/2311.08666/) (2023)
- [The Hidden Cost of Using Amazon Mechanical Turk for Research](https://mlsanity.com/paper/2101.04459/) (2021)
- [Cura: Curation at Social Media Scale](https://mlsanity.com/paper/2308.13841/) (2023)

## Abstract (arXiv)

Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction.