mlsanityData to 14 Oct 2026

Task Decomposition for Efficient Annotation

arXiv 2606.24734 ↗First posted 23 Jun 2026cs.CL cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

High-quality annotations of structured representations are expensive to collect over large corpora. Manual annotation of structure is laborious, and model-based annotation, although cheaper to generate, requires expensive validation and potentially significant supervision to ensure that the annotation quality is strong enough to be useful downstream. In traditional annotation workflows, annotation of each complete example is performed end-to-end by a single annotator. However, structured annotation is complex, and each aspect of the task represents a unique challenge with an associated inferential load for a given annotator. Modern annotation projects can incorporate heterogeneous groups of annotators, including both models and human annotators with varying domain and linguistic expertise. It remains unclear, however, how to redesign annotation tasks in this setting, where efforts are discriminately allocated across heterogeneous annotators with respect to distinct annotation challenges. We propose to decompose annotation tasks into sub-tasks in order to reduce the aggregate inferential load of annotation projects. Inspired by the notion of centers from centering theory, we introduce a formal model of inferential load based on the degrees of freedom in the space of valid annotations. Using this model, we show that identifying these centers (i.e. salient anchor entities realized by annotation sub-tasks) constrains the output space complexity, and decompositions which isolate and advance center identification reduce the aggregate inferential load. We provide guidelines for decomposing complex structured annotation tasks, supported by examples demonstrating improved cost-efficiency from our prior work. Finally, we present a procedure for allocating sub-tasks across annotators to maximize quality under a fixed budget.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 67 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

29 of 67 references.

References · 29

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  4. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  5. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  6. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  7. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 119
  8. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
  9. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  10. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  11. 2023LLMs Accelerate Annotation for Medical Information Extractioncited by 77
  12. 2018Large-Scale QA-SRL Parsingcited by 45
17 more
  1. 2024NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Datacited by 34
  2. 2019Sequence-to-Nuggets: Nested Entity Mention Detection via Anchor-Region Networkscited by 33
  3. 2018Syntactic Scaffolds for Semantic Structurescited by 27
  4. 2020Unsupervised Label-aware Event Trigger and Argument Classificationcited by 27
  5. 2023Are Large Language Models Robust Coreference Resolvers?cited by 15
  6. 2025Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreementcited by 14
  7. 2024Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectivescited by 13
  8. 2018Learning Joint Semantic Parsers from Disjoint Datacited by 12
  9. 2017Deep Multitask Learning for Semantic Dependency Parsingcited by 10
  10. 2019Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extractioncited by 9
  11. 2024Major Entity Identification: A Generalizable Alternative to Coreference Resolutioncited by 9
  12. 2025Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segmentscited by 9
  13. 2022Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolutioncited by 6
  14. 2023ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexitycited by 3
  15. 2024Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Datacited by 3
  16. 2023Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Trainingcited by 2
  17. 2023The Case for Scalable, Data-Driven Theory: A Paradigm for Scientific Progress in NLPcited by 2

Built on by

Not cited yet.