# Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models

arXiv: https://arxiv.org/abs/2403.16851 · HTML: https://mlsanity.com/paper/2403.16851/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 95 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective](https://mlsanity.com/paper/1401.4321/) (2014)
- [Scholarly use of social media and altmetrics: a review of the literature](https://mlsanity.com/paper/1608.08112/) (2016)
- [Scholarly Twitter metrics](https://mlsanity.com/paper/1806.02201/) (2018)
- [Dynamics of Cross-Platform Attention to Retracted Papers](https://mlsanity.com/paper/2110.07798/) (2021)
- [Differences in Personal and Professional Tweets of Scholars](https://mlsanity.com/paper/1505.01515/) (2015)
- [Allegation of scientific misconduct increases Twitter attention](https://mlsanity.com/paper/1802.00606/) (2018)
- [Online Engagement with Retracted Articles: Who, When, and How?](https://mlsanity.com/paper/2203.04228/) (2022)

## Abstract (arXiv)

Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analysed 3,815 tweets referencing 604 retracted articles and 3,373 tweets referencing 668 comparable non-retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non-retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post-publication discourse should be applied with caution. A human-AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.