# A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Models

arXiv: https://arxiv.org/abs/2504.08961 · HTML: https://mlsanity.com/paper/2504.08961/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 10 of 10)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant](https://mlsanity.com/paper/2403.11299/) (2024)
- [Dialog Acts for Task-Driven Embodied Agents](https://mlsanity.com/paper/2209.12953/) (2022)
- [FullAnno: A Data Engine for Enhancing Image Comprehension of MLLMs](https://mlsanity.com/paper/2409.13540/) (2024)
- [Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue](https://mlsanity.com/paper/2311.07445/) (2023)
- [Gunrock 2.0: A User Adaptive Social Conversational System](https://mlsanity.com/paper/2011.08906/) (2020)
- [Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4](https://mlsanity.com/paper/2407.04130/) (2024)
- [Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting](https://mlsanity.com/paper/2409.09615/) (2024)

## Built on by (most cited first, 1 of 1)

- [Leveraging Large Language Models to Identify Conversation Threads in Collaborative Learning](https://mlsanity.com/paper/2510.22844/) (2025)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have shown promise in automating discourse annotation for conversations. While manually designing tree annotation schemes significantly improves annotation quality for humans and models, their creation remains time-consuming and requires expert knowledge. We propose a fully automated pipeline that uses LLMs to construct such schemes and perform annotation. We evaluate our approach on speech functions (SFs) and the Switchboard-DAMSL (SWBD-DAMSL) taxonomies. Our experiments compare various design choices, and we show that frequency-guided decision trees, paired with an advanced LLM for annotation, can outperform previously manually designed trees and even match or surpass human annotators while significantly reducing the time required for annotation. We release all code and resultant schemes and annotations to facilitate future research on discourse annotation.