# POTATO: The Portable Text Annotation Tool

arXiv: https://arxiv.org/abs/2212.08620 · HTML: https://mlsanity.com/paper/2212.08620/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [The Ethics of AI Ethics -- An Evaluation of Guidelines](https://mlsanity.com/paper/1903.03425/) (2019)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Masakhane -- Machine Translation For Africa](https://mlsanity.com/paper/2003.11529/) (2020)
- [Modeling Framing in Immigration Discourse on Social Media](https://mlsanity.com/paper/2104.06443/) (2021)
- [Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications](https://mlsanity.com/paper/2109.14776/) (2021)
- [Quantifying Intimacy in Language](https://mlsanity.com/paper/2011.03020/) (2020)
- [Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations](https://mlsanity.com/paper/2102.08368/) (2021)
- [LightTag: Text Annotation Platform](https://mlsanity.com/paper/2109.02320/) (2021)
- [SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis](https://mlsanity.com/paper/2210.01108/) (2022)
- [Modeling Information Change in Science Communication with Semantically Matched Paraphrases](https://mlsanity.com/paper/2210.13001/) (2022)
- [An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog](https://mlsanity.com/paper/2109.12212/) (2021)

## Built on by (most cited first, 30 of 64)

- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [Human Feedback is not Gold Standard](https://mlsanity.com/paper/2309.16349/) (2023)
- [An Emulator for Fine-Tuning Large Language Models using Small Language Models](https://mlsanity.com/paper/2310.12962/) (2023)
- [Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks](https://mlsanity.com/paper/2310.13800/) (2023)
- [When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset](https://mlsanity.com/paper/2306.06826/) (2023)
- [Modifying Large Language Model Post-Training for Diverse Creative Writing](https://mlsanity.com/paper/2503.17126/) (2025)
- [Chinese Open Instruction Generalist: A Preliminary Release](https://mlsanity.com/paper/2304.07987/) (2023)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback](https://mlsanity.com/paper/2408.15549/) (2024)
- [BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages](https://mlsanity.com/paper/2502.11926/) (2025)
- [SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes](https://mlsanity.com/paper/2403.07726/) (2024)
- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoral](https://mlsanity.com/paper/2502.14083/) (2025)
- [ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments](https://mlsanity.com/paper/2502.19852/) (2025)
- [CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding](https://mlsanity.com/paper/2503.10688/) (2025)
- [Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs](https://mlsanity.com/paper/2412.15993/) (2024)
- [Pearmut: Human Evaluation of Translation Made Trivial](https://mlsanity.com/paper/2601.02933/) (2026)
- [A Reality Check on Context Utilisation for Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.17031/) (2024)
- [Understanding Fine-grained Distortions in Reports of Scientific Findings](https://mlsanity.com/paper/2402.12431/) (2024)
- [DMOps: Data Management Operation and Recipes](https://mlsanity.com/paper/2301.01228/) (2023)
- [The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change](https://mlsanity.com/paper/2311.12664/) (2023)
- [M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset](https://mlsanity.com/paper/2510.23508/) (2025)
- [Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships](https://mlsanity.com/paper/2307.02763/) (2023)
- [Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation](https://mlsanity.com/paper/2308.06953/) (2023)
- [Mapping the Podcast Ecosystem with the Structured Podcast Research Corpus](https://mlsanity.com/paper/2411.07892/) (2024)
- [Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols](https://mlsanity.com/paper/2501.13284/) (2025)
- [What is Stigma Attributed to? A Theory-Grounded, Expert-Annotated Interview Corpus for Demystifying Mental-Health Stigma](https://mlsanity.com/paper/2505.12727/) (2025)

## Abstract (arXiv)

We present POTATO, the Portable text annotation tool, a free, fully open-sourced annotation system that 1) supports labeling many types of text and multimodal data; 2) offers easy-to-configure features to maximize the productivity of both deployers and annotators (convenient templates for common ML/NLP tasks, active learning, keypress shortcuts, keyword highlights, tooltips); and 3) supports a high degree of customization (editable UI, inserting pre-screening questions, attention and qualification tests). Experiments over two annotation tasks suggest that POTATO improves labeling speed through its specially-designed productivity features, especially for long documents and complex tasks. POTATO is available at https://github.com/davidjurgens/potato and will continue to be updated.