USDC: A Dataset of underlineUser underlineStance and underlineDogmatism in Long underlineConversations
Not on the map. Why?
Abstract (arXiv)
Analyzing user opinion changes in long conversation threads is extremely critical for applications like enhanced personalization, market research, political campaigns, customer service, targeted advertising, and content moderation. Unfortunately, previous studies on stance and dogmatism in user conversations have focused on training models using datasets annotated at the post level, treating each post as independent and randomly sampling posts from conversation threads. Hence, first, we build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC. USDC contains annotations for 2 tasks: i) User Stance classification, which involves labeling a user's stance in a post within a conversation on a five-point scale; ii) User Dogmatism classification, which involves labeling a user's overall opinion in the conversation on a four-point scale. Besides being time-consuming and costly, manual annotations for USDC are challenging because: 1) Conversation threads could be very long, increasing the chances of noisy annotations; and 2) Interpreting instances where a user changes their opinion within a conversation is difficult because often such transitions are subtle and not expressed explicitly. Hence, we leverage majority voting on zero-shot, one-shot, and few-shot annotations from Mistral Large and GPT-4 to automate the annotation process. Human annotations on 200 test conversations achieved inter-annotator agreement scores of 0.49 for stance and 0.50 for dogmatism with these LLM annotations, indicating a reasonable level of consistency between human and LLM annotations. USDC is then used to finetune and instruction-tune multiple deployable small language models like LLaMA, Falcon and Vicuna for the stance and dogmatism classification tasks. We make the code and dataset publicly available [https://github.com/mounikamarreddy/USDC].
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 56 references that are in the map. First posted on arXiv 24 Jun 2024.
Builds on
32 of 56 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2024Mixtral of Expertscited by 1,785
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
20 more
- 2020Recipes for building an open-domain chatbotcited by 593
- 2023The Falcon Series of Open Language Modelscited by 526
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
- 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
- 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
- 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
- 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
- 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
- 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
- 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
- 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
- 2023Making Large Language Models Better Data Creatorscited by 28
- 2024A Challenge Dataset and Effective Models for Conversational Stance Detectioncited by 17
- 2022Improved Target-specific Stance Detection on Social Media Platforms by Delving into Conversation Threadscited by 16
- 2020Stance in Replies and Quotes (SRQ): A New Dataset For Learning Stance in Twitter Conversationscited by 14
- 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
- 2022Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinioncited by 3
- 2023Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictionscited by 2
- 2016Identifying Dogmatism in Social Media: Signals and Modelscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.