# Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media

arXiv: https://arxiv.org/abs/2502.01991 · HTML: https://mlsanity.com/paper/2502.01991/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 82 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Identifying Morality Frames in Political Tweets using Relational Learning](https://mlsanity.com/paper/2109.04535/) (2021)
- [A Holistic Framework for Analyzing the COVID-19 Vaccine Debate](https://mlsanity.com/paper/2205.01817/) (2022)
- [Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation](https://mlsanity.com/paper/2401.10838/) (2024)
- [Modeling Content and Context with Deep Relational Learning](https://mlsanity.com/paper/2010.10453/) (2020)
- [Interactive Concept Learning for Uncovering Latent Themes in Large Text Collections](https://mlsanity.com/paper/2305.05094/) (2023)
- [Polarization of the Vaccination Debate on Facebook](https://mlsanity.com/paper/1801.02903/) (2018)
- [Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision](https://mlsanity.com/paper/2210.10031/) (2022)
- [Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy](https://mlsanity.com/paper/2404.10259/) (2024)
- [Weakly Supervised Learning for Analyzing Political Campaigns on Facebook](https://mlsanity.com/paper/2210.10669/) (2022)
- [Discovering Latent Themes in Social Media Messaging: A Machine-in-the-Loop Approach Integrating LLMs](https://mlsanity.com/paper/2403.10707/) (2024)
- [Towards Few-Shot Identification of Morality Frames using In-Context Learning](https://mlsanity.com/paper/2302.02029/) (2023)
- [Moral Narratives Around the Vaccination Debate on Facebook](https://mlsanity.com/paper/2206.01598/) (2022)
- [From Hesitancy Framings to Vaccine Hesitancy Profiles: A Journey of Stance, Ontological Commitments and Moral Foundations](https://mlsanity.com/paper/2202.09456/) (2022)
- [Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation](https://mlsanity.com/paper/2410.05401/) (2024)
- [Analysis of Climate Campaigns on Social Media using Bayesian Model Averaging](https://mlsanity.com/paper/2305.06174/) (2023)

## Built on by (most cited first, 1 of 1)

- [Iterative Topic Taxonomy Induction with LLMs: A Case Study of Electoral Advertising](https://mlsanity.com/paper/2510.15125/) (2025)

## Abstract (arXiv)

Nowadays, social media is pivotal in shaping public discourse, especially on polarizing issues like vaccination, where diverse moral perspectives influence individual opinions. In NLP, data scarcity and complexity of psycholinguistic tasks, such as identifying morality frames, make relying solely on human annotators costly, time-consuming, and prone to inconsistency due to cognitive load. To address these issues, we leverage large language models (LLMs), which are adept at adapting new tasks through few-shot learning, utilizing a handful of in-context examples coupled with explanations that connect examples to task principles. Our research explores LLMs' potential to assist human annotators in identifying morality frames within vaccination debates on social media. We employ a two-step process: generating concepts and explanations with LLMs, followed by human evaluation using a "think-aloud" tool. Our study shows that integrating LLMs into the annotation process enhances accuracy, reduces task difficulty, lowers cognitive load, suggesting a promising avenue for human-AI collaboration in complex psycholinguistic tasks.