# Quality and Efficiency of Manual Annotation: Pre-annotation Bias

arXiv: https://arxiv.org/abs/2306.09307 · HTML: https://mlsanity.com/paper/2306.09307/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [SciDTB: Discourse Dependency TreeBank for Scientific Abstracts](https://mlsanity.com/paper/1806.03653/) (2018)
- [Prague Dependency Treebank -- Consolidated 1.0](https://mlsanity.com/paper/2006.03679/) (2020)
- [BKTreebank: Building a Vietnamese Dependency Treebank](https://mlsanity.com/paper/1710.05519/) (2017)
- [Creating a Universal Dependencies Treebank of Spoken Frisian-Dutch Code-switched Data](https://mlsanity.com/paper/2102.11152/) (2021)

## Built on by (most cited first, 8 of 8)

- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [eRST: A Signaled Graph Theory of Discourse Relations and Organization](https://mlsanity.com/paper/2403.13560/) (2024)
- [Bias in the Loop: How Humans Evaluate AI-Generated Suggestions](https://mlsanity.com/paper/2509.08514/) (2025)
- [A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction](https://mlsanity.com/paper/2602.04320/) (2026)
- [From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning](https://mlsanity.com/paper/2510.24528/) (2025)
- [CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection](https://mlsanity.com/paper/2604.04174/) (2026)
- [TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech](https://mlsanity.com/paper/2606.01820/) (2026)
- [Prague Dependency Treebank -- Consolidated 2.0: Enriching a Complex Annotation Scheme](https://mlsanity.com/paper/2606.24324/) (2026)

## Abstract (arXiv)

This paper presents an analysis of annotation using an automatic pre-annotation for a mid-level annotation complexity task -- dependency syntax annotation. It compares the annotation efforts made by annotators using a pre-annotated version (with a high-accuracy parser) and those made by fully manual annotation. The aim of the experiment is to judge the final annotation quality when pre-annotation is used. In addition, it evaluates the effect of automatic linguistically-based (rule-formulated) checks and another annotation on the same data available to the annotators, and their influence on annotation quality and efficiency. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality.