# Leveraging Large Language Models (LLMs) to Support Collaborative Human-AI Online Risk Data Annotation

arXiv: https://arxiv.org/abs/2404.07926 · HTML: https://mlsanity.com/paper/2404.07926/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 38 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 7 of 7)

- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://mlsanity.com/paper/2212.14548/) (2022)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?](https://mlsanity.com/paper/1912.08320/) (2019)
- [Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction](https://mlsanity.com/paper/1909.07065/) (2019)
- [QualiGPT: GPT as an easy-to-use tool for qualitative coding](https://mlsanity.com/paper/2310.07061/) (2023)

## Built on by (most cited first, 2 of 2)

- [Estimating Contribution Quality in Online Deliberations Using a Large Language Model](https://mlsanity.com/paper/2408.11936/) (2024)
- [DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production](https://mlsanity.com/paper/2412.08069/) (2024)

## Abstract (arXiv)

In this position paper, we discuss the potential for leveraging LLMs as interactive research tools to facilitate collaboration between human coders and AI to effectively annotate online risk data at scale. Collaborative human-AI labeling is a promising approach to annotating large-scale and complex data for various tasks. Yet, tools and methods to support effective human-AI collaboration for data annotation are under-studied. This gap is pertinent because co-labeling tasks need to support a two-way interactive discussion that can add nuance and context, particularly in the context of online risk, which is highly subjective and contextualized. Therefore, we provide some of the early benefits and challenges of using LLMs-based tools for risk annotation and suggest future directions for the HCI research community to leverage LLMs as research tools to facilitate human-AI collaboration in contextualized online data annotation. Our research interests align very well with the purposes of the LLMs as Research Tools workshop to identify ongoing applications and challenges of using LLMs to work with data in HCI research. We anticipate learning valuable insights from organizers and participants into how LLMs can help reshape the HCI community's methods for working with data.