# Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation

arXiv: https://arxiv.org/abs/2601.12061 · HTML: https://mlsanity.com/paper/2601.12061/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 17 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring](https://mlsanity.com/paper/2106.06719/) (2021)
- [Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study](https://mlsanity.com/paper/2305.08391/) (2023)
- [Improving Context Modeling in Neural Topic Segmentation](https://mlsanity.com/paper/2010.03138/) (2020)
- [Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations](https://mlsanity.com/paper/2305.14556/) (2023)
- [Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation](https://mlsanity.com/paper/2305.02747/) (2023)
- [SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation](https://mlsanity.com/paper/2305.08371/) (2023)
- [Intent Matters: Enhancing AI Tutoring with Fine-Grained Pedagogical Intent Annotation](https://mlsanity.com/paper/2506.07626/) (2025)
- [Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation](https://mlsanity.com/paper/2507.13138/) (2025)
- [Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations](https://mlsanity.com/paper/2411.07598/) (2024)
- [Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications](https://mlsanity.com/paper/2507.00543/) (2025)
- [AI Annotation Orchestration: Evaluating LLM verifiers to Improve the Quality of LLM Annotations in Learning Analytics](https://mlsanity.com/paper/2511.09785/) (2025)
- [LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding](https://mlsanity.com/paper/2504.19734/) (2025)

## Abstract (arXiv)

Dialogue Act (DA) annotation typically treats communicative or pedagogical intent as localized to individual utterances or turns. This leads annotators to agree on the underlying action while disagreeing on segment boundaries, reducing apparent reliability. We propose codebook-injected segmentation, which conditions boundary decisions on downstream annotation criteria, and evaluate LLM-based segmenters against standard and retrieval-augmented baselines. To assess these without gold labels, we introduce evaluation metrics for span consistency, distinctiveness, and human-AI distributional agreement. We found DA-awareness produces segments that are internally more consistent than text-only baselines. While LLMs excel at creating construct-consistent spans, coherence-based baselines remain superior at detecting global shifts in dialogue flow. Across two datasets, no single segmenter dominates. Improvements in within-segment coherence frequently trade off against boundary distinctiveness and human-AI distributional agreement. These results highlight segmentation as a consequential design choice that should be optimized for downstream objectives rather than a single performance score.