CALICO: A Human-Centered, Codebook-Aligned System for Annotation

arXiv 2609.14726 ↗First posted 13 Sep 2026cs.HC cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 35 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

16 of 35 references.

References · 16

2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2023Large Language Models as Optimizerscited by 830
2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
2023Automated Annotation with Generative AI Requires Validationcited by 75
2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 43
2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 28

4 more

2025Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Taskscited by 21
2024The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detectioncited by 14
2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation Systemcited by 5
2025Co-DETECT: Collaborative Discovery of Edge Cases in Text Classificationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way