# A Retrospective Analysis of the Fake News Challenge Stance Detection Task

arXiv: https://arxiv.org/abs/1806.05180 · HTML: https://mlsanity.com/paper/1806.05180/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 30 of 49)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [Cross-Domain Label-Adaptive Stance Detection](https://mlsanity.com/paper/2104.07467/) (2021)
- [Overview of the CLEF--2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News](https://mlsanity.com/paper/2109.12987/) (2021)
- [Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic](https://mlsanity.com/paper/2211.10057/) (2022)
- [Interpretable Propaganda Detection in News Articles](https://mlsanity.com/paper/2108.12802/) (2021)
- [Topic-Aware Evidence Reasoning and Stance-Aware Aggregation for Fact Verification](https://mlsanity.com/paper/2106.01191/) (2021)
- [Investigating label suggestions for opinion mining in German Covid-19 social media](https://mlsanity.com/paper/2105.12980/) (2021)
- [GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates](https://mlsanity.com/paper/2202.08175/) (2022)
- [The CLEF-2025 CheckThat! Lab: Subjectivity, Fact-Checking, Claim Normalization, and Retrieval](https://mlsanity.com/paper/2503.14828/) (2025)
- [Misinformation Detection on YouTube Using Video Captions](https://mlsanity.com/paper/2107.00941/) (2021)
- [MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification](https://mlsanity.com/paper/2109.06416/) (2021)
- [Dataset of Fake News Detection and Fact Verification: A Survey](https://mlsanity.com/paper/2111.03299/) (2021)
- [Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims](https://mlsanity.com/paper/2204.12294/) (2022)
- [Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study](https://mlsanity.com/paper/2110.07693/) (2021)
- [STEM: Unsupervised STructural EMbedding for Stance Detection](https://mlsanity.com/paper/2112.00712/) (2021)
- [AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking](https://mlsanity.com/paper/2104.13559/) (2021)
- [Stanceosaurus: Classifying Stance Towards Multilingual Misinformation](https://mlsanity.com/paper/2210.15954/) (2022)
- [A Graph-Based Context-Aware Model to Understand Online Conversations](https://mlsanity.com/paper/2211.09207/) (2022)
- [Nothing Stands Alone: Relational Fake News Detection with Hypergraph Neural Networks](https://mlsanity.com/paper/2212.12621/) (2022)
- [TIB's Visual Analytics Group at MediaEval '20: Detecting Fake News on Corona Virus and 5G Conspiracy](https://mlsanity.com/paper/2101.03529/) (2021)
- [Applying Automatic Text Summarization for Fake News Detection](https://mlsanity.com/paper/2204.01841/) (2022)
- [The COVMis-Stance dataset: Stance Detection on Twitter for COVID-19 Misinformation](https://mlsanity.com/paper/2204.02000/) (2022)
- [Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets](https://mlsanity.com/paper/2205.11472/) (2022)
- [Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection](https://mlsanity.com/paper/2306.00765/) (2023)
- [Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals](https://mlsanity.com/paper/2408.07453/) (2024)
- [All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection](https://mlsanity.com/paper/2601.04160/) (2026)
- [WhatTheWikiFact: Fact-Checking Claims Against Wikipedia](https://mlsanity.com/paper/2105.00826/) (2021)
- [The CLEF-2026 CheckThat! Lab: Advancing Multilingual Fact-Checking](https://mlsanity.com/paper/2602.09516/) (2026)
- [Tribrid: Stance Classification with Neural Inconsistency Detection](https://mlsanity.com/paper/2109.06508/) (2021)

## Abstract (arXiv)

The 2017 Fake News Challenge Stage 1 (FNC-1) shared task addressed a stance classification task as a crucial first step towards detecting fake news. To date, there is no in-depth analysis paper to critically discuss FNC-1's experimental setup, reproduce the results, and draw conclusions for next-generation stance classification methods. In this paper, we provide such an in-depth analysis for the three top-performing systems. We first find that FNC-1's proposed evaluation metric favors the majority class, which can be easily classified, and thus overestimates the true discriminative power of the methods. Therefore, we propose a new F1-based metric yielding a changed system ranking. Next, we compare the features and architectures used, which leads to a novel feature-rich stacked LSTM model that performs on par with the best systems, but is superior in predicting minority classes. To understand the methods' ability to generalize, we derive a new dataset and perform both in-domain and cross-domain experiments. Our qualitative and quantitative study helps interpreting the original FNC-1 scores and understand which features help improving performance and why. Our new dataset and all source code used during the reproduction study are publicly available for future research.