# Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

arXiv: https://arxiv.org/abs/2403.00236 · HTML: https://mlsanity.com/paper/2403.00236/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 72 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://mlsanity.com/paper/2010.03640/) (2020)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://mlsanity.com/paper/2212.14548/) (2022)
- [Stance and Sentiment in Tweets](https://mlsanity.com/paper/1605.01655/) (2016)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://mlsanity.com/paper/2205.00619/) (2022)
- [Detecting Stance in Media on Global Warming](https://mlsanity.com/paper/2010.15149/) (2020)
- [X-Stance: A Multilingual Multi-Target Dataset for Stance Detection](https://mlsanity.com/paper/2003.08385/) (2020)

## Built on by (most cited first, 6 of 6)

- [LLM Chain Ensembles for Scalable and Accurate Data Annotation](https://mlsanity.com/paper/2410.13006/) (2024)
- [Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness](https://mlsanity.com/paper/2411.08977/) (2024)
- [Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models](https://mlsanity.com/paper/2410.19195/) (2024)
- [What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?](https://mlsanity.com/paper/2511.18616/) (2025)
- [Zero-shot Tweet-Level Stance Detection Enhanced by External Knowledge and Reflective Chain-of-Thought Reasoning](https://mlsanity.com/paper/2606.26571/) (2026)
- [MoPrO-SD: Modular Prompt Optimization for Stance Detection](https://mlsanity.com/paper/2610.13386/) (2026)

## Abstract (arXiv)

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations under different prompts and decoding strategies, as well as the potential biases of the model. We show that the zero-shot approach can match or outperform state-of-the-art benchmarks, including fine-tuned models. We provide various insights into its performance including the sensitivity to instructions and prompts, the decoding strategies, the perplexity of the prompts, and to negations and oppositions present in prompts. Finally, we ensure that the LLM has not been trained on test datasets, and identify a positivity bias which may partially explain the performance differences across decoding strategie