# Contextual information integration for stance detection via cross-attention

arXiv: https://arxiv.org/abs/2211.01874 · HTML: https://mlsanity.com/paper/2211.01874/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 82 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://mlsanity.com/paper/2010.03640/) (2020)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [RumourEval 2019: Determining Rumour Veracity and Support for Rumours](https://mlsanity.com/paper/1809.06683/) (2018)
- [Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims](https://mlsanity.com/paper/1906.03538/) (2019)
- [A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking](https://mlsanity.com/paper/1911.01214/) (2019)
- [Cross-topic Argument Mining from Heterogeneous Sources Using Attention-based Neural Networks](https://mlsanity.com/paper/1802.05758/) (2018)
- [Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers](https://mlsanity.com/paper/2005.11787/) (2020)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://mlsanity.com/paper/1708.01425/) (2017)
- [Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter](https://mlsanity.com/paper/2005.00388/) (2020)
- [Cross-Target Stance Classification with Self-Attention Networks](https://mlsanity.com/paper/1805.06593/) (2018)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [Adversarial Learning for Zero-Shot Stance Detection on Social Media](https://mlsanity.com/paper/2105.06603/) (2021)
- [Stance Detection Benchmark: How Robust Is Your Stance Detection?](https://mlsanity.com/paper/2001.01565/) (2020)

## Built on by (most cited first, 6 of 6)

- [The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification](https://mlsanity.com/paper/2409.17929/) (2024)
- [Cross-Target Stance Detection: A Survey of Techniques, Datasets, and Challenges](https://mlsanity.com/paper/2409.13594/) (2024)
- [Putting Context in Context: the Impact of Discussion Structure on Text Classification](https://mlsanity.com/paper/2402.02975/) (2024)
- [Enabling Contextual Soft Moderation on Social Media through Contrastive Textual Deviation](https://mlsanity.com/paper/2407.20910/) (2024)
- [Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study](https://mlsanity.com/paper/2404.03987/) (2024)
- [Contextualized Prompting For Stance Detection On Social Media](https://mlsanity.com/paper/2606.06022/) (2026)

## Abstract (arXiv)

Stance detection deals with identifying an author's stance towards a target. Most existing stance detection models are limited because they do not consider relevant contextual information which allows for inferring the stance correctly. Complementary context can be found in knowledge bases but integrating the context into pretrained language models is non-trivial due to the graph structure of standard knowledge bases. To overcome this, we explore an approach to integrate contextual information as text which allows for integrating contextual information from heterogeneous sources, such as structured knowledge sources and by prompting large language models. Our approach can outperform competitive baselines on a large and diverse stance detection benchmark in a cross-target setup, i.e. for targets unseen during training. We demonstrate that it is more robust to noisy context and can regularize for unwanted correlations between labels and target-specific vocabulary. Finally, it is independent of the pretrained language model in use.