# MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

arXiv: https://arxiv.org/abs/2109.08113 · HTML: https://mlsanity.com/paper/2109.08113/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Stance Detection Benchmark: How Robust Is Your Stance Detection?](https://mlsanity.com/paper/2001.01565/) (2020)
- [MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection](https://mlsanity.com/paper/1606.03784/) (2016)
- [Your Stance is Exposed! Analysing Possible Factors for Stance Detection on Social Media](https://mlsanity.com/paper/1908.03146/) (2019)
- [Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality](https://mlsanity.com/paper/2105.03484/) (2021)

## Built on by (most cited first, 11 of 11)

- [Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation](https://mlsanity.com/paper/2306.01183/) (2023)
- [Large Human Language Models: A Need and the Challenges](https://mlsanity.com/paper/2312.07751/) (2023)
- [Stanceosaurus: Classifying Stance Towards Multilingual Misinformation](https://mlsanity.com/paper/2210.15954/) (2022)
- [SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks](https://mlsanity.com/paper/2402.01980/) (2024)
- [Human Language Modeling](https://mlsanity.com/paper/2205.05128/) (2022)
- [Debiasing Stance Detection Models with Counterfactual Reasoning and Adversarial Bias Learning](https://mlsanity.com/paper/2212.10392/) (2022)
- [Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?](https://mlsanity.com/paper/2401.12492/) (2024)
- [From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP](https://mlsanity.com/paper/2601.07988/) (2026)
- [Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks](https://mlsanity.com/paper/2503.00124/) (2025)
- [Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment](https://mlsanity.com/paper/2112.13795/) (2021)
- [Addressing the Ecological Fallacy in Larger LMs with Human Context](https://mlsanity.com/paper/2603.05928/) (2026)

## Abstract (arXiv)

Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher-levels of context (i.e., sequences of messages) is under-explored. In stance detection and other social media tasks where the goal is to predict an attribute of a message, we have contextual data that is loosely semantically connected by authorship. Here, we introduce Message-Level Transformer (MeLT) -- a hierarchical message-encoder pre-trained over Twitter and applied to the task of stance prediction. We focus on stance prediction as a task benefiting from knowing the context of the message (i.e., the sequence of previous messages). The model is trained using a variant of masked-language modeling; where instead of predicting tokens, it seeks to generate an entire masked (aggregated) message vector via reconstruction loss. We find that applying this pre-trained masked message-level transformer to the downstream task of stance detection achieves F1 performance of 67%.