# Discovering Latent Themes in Social Media Messaging: A Machine-in-the-Loop Approach Integrating LLMs arXiv: https://arxiv.org/abs/2403.10707 ยท HTML: https://mlsanity.com/paper/2403.10707/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 62 references that are in the map. - First posted on arXiv 15 Mar 2024. ## Builds on (most cited first, 15 of 15) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022) - [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023) - [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023) - [A Holistic Framework for Analyzing the COVID-19 Vaccine Debate](https://mlsanity.com/paper/2205.01817/) (2022) - [Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model](https://mlsanity.com/paper/2305.13014/) (2023) - [Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook](https://mlsanity.com/paper/2001.10581/) (2020) - [Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision](https://mlsanity.com/paper/2210.10031/) (2022) - [Clandestino or Rifugiato? Anti-immigration Facebook Ad Targeting in Italy](https://mlsanity.com/paper/2103.09224/) (2021) - [Facebook Ads: Politics of Migration in Italy](https://mlsanity.com/paper/2010.04458/) (2020) ## Built on by (most cited first, 8 of 8) - [Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy](https://mlsanity.com/paper/2404.10259/) (2024) - [Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation](https://mlsanity.com/paper/2410.05401/) (2024) - [LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages](https://mlsanity.com/paper/2412.10918/) (2024) - [Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media](https://mlsanity.com/paper/2502.01991/) (2025) - [Iterative Topic Taxonomy Induction with LLMs: A Case Study of Electoral Advertising](https://mlsanity.com/paper/2510.15125/) (2025) - [HAMLET: Healthcare-focused Adaptive Multilingual Learning Embedding-based Topic Modeling](https://mlsanity.com/paper/2505.07157/) (2025) - [Paid Voices vs. Public Feeds: Interpretable Cross-Platform Theme-Based Analysis of Climate Discourse](https://mlsanity.com/paper/2601.13317/) (2026) - [Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation](https://mlsanity.com/paper/2604.09626/) (2026) ## Abstract (arXiv) Grasping the themes of social media content is key to understanding the narratives that influence public opinion and behavior. The thematic analysis goes beyond traditional topic-level analysis, which often captures only the broadest patterns, providing deeper insights into specific and actionable themes such as "public sentiment towards vaccination", "political discourse surrounding climate policies," etc. In this paper, we introduce a novel approach to uncovering latent themes in social media messaging. Recognizing the limitations of the traditional topic-level analysis, which tends to capture only overarching patterns, this study emphasizes the need for a finer-grained, theme-focused exploration. Traditional theme discovery methods typically involve manual processes and a human-in-the-loop approach. While valuable, these methods face challenges in scalability, consistency, and resource intensity in terms of time and cost. To address these challenges, we propose a machine-in-the-loop approach that leverages the advanced capabilities of Large Language Models (LLMs). To demonstrate our approach, we apply our framework to contentious topics, such as climate debate and vaccine debate. We use two publicly available datasets: (1) the climate campaigns dataset of 21k Facebook ads and (2) the COVID-19 vaccine campaigns dataset of 9k Facebook ads. Our quantitative and qualitative analysis shows that our methodology yields more accurate and interpretable results compared to the baselines. Our results not only demonstrate the effectiveness of our approach in uncovering latent themes but also illuminate how these themes are tailored for demographic targeting in social media contexts. Additionally, our work sheds light on the dynamic nature of social media, revealing the shifts in the thematic focus of messaging in response to real-world events.