Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification
Not on the map. Why?
Abstract (arXiv)
We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models (LLMs). Co-DETECT starts with an initial, sketch-level codebook and dataset provided by a domain expert, then leverages the LLM to annotate the data and identify edge cases that are not well described by the initial codebook. Specifically, Co-DETECT flags challenging examples, induces high-level, generalizable descriptions of edge cases, and assists user in incorporating edge case handling rules to improve the codebook. This iterative process enables more effective handling of nuanced phenomena through compact, generalizable annotation rules. Extensive user study, qualitative and quantitative analyses prove the effectiveness of Co-DETECT.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 7 Jul 2025.
Builds on
21 of 43 references.
References · 21
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
- 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
- 2022Is GPT-3 a Good Data Annotator?cited by 208
- 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
- 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
- 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
- 2023Automated Annotation with Generative AI Requires Validationcited by 76
- 2023Large Language Models Enable Few-Shot Clusteringcited by 72
- 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 70
- 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 62
9 more
- 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
- 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
- 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
- 2024Best Practices for Text Annotation with Large Language Modelscited by 36
- 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 30
- 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
- 2023Using Large Language Models to Support Thematic Analysis in Empirical Legal Studiescited by 25
- 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
- 2025SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Interventioncited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.