# Effects of Collaboration on the Performance of Interactive Theme Discovery Systems

arXiv: https://arxiv.org/abs/2408.09030 · HTML: https://mlsanity.com/paper/2408.09030/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Modeling Framing in Immigration Discourse on Social Media](https://mlsanity.com/paper/2104.06443/) (2021)
- [CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models](https://mlsanity.com/paper/2304.07366/) (2023)
- [Identifying Morality Frames in Political Tweets using Relational Learning](https://mlsanity.com/paper/2109.04535/) (2021)
- [CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis](https://mlsanity.com/paper/2304.05560/) (2023)
- [A Holistic Framework for Analyzing the COVID-19 Vaccine Debate](https://mlsanity.com/paper/2205.01817/) (2022)
- [Interactive Concept Learning for Uncovering Latent Themes in Large Text Collections](https://mlsanity.com/paper/2305.05094/) (2023)
- [A Query-Driven Topic Model](https://mlsanity.com/paper/2106.07346/) (2021)
- [A User-Centered, Interactive, Human-in-the-Loop Topic Modelling System](https://mlsanity.com/paper/2304.01774/) (2023)
- [Analysis of Climate Campaigns on Social Media using Bayesian Model Averaging](https://mlsanity.com/paper/2305.06174/) (2023)

## Abstract (arXiv)

NLP-assisted solutions to support qualitative data analysis have gained considerable traction. However, no unified evaluation framework exists which can account for the many different settings in which qualitative researchers may employ them. In this paper, we propose a framework to evaluate the way collaboration settings may produce different research outcomes across a variety of interactive systems. Specifically, we study the impact of synchronous vs. asynchronous collaboration using three different NLP-assisted qualitative research tools and present a comprehensive analysis of the differences in the consistency, cohesiveness, and correctness of their outcomes.