# It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments

arXiv: https://arxiv.org/abs/2404.18832 · HTML: https://mlsanity.com/paper/2404.18832/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [A Fine-Grained Sentiment Dataset for Norwegian](https://mlsanity.com/paper/1911.12722/) (2019)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Entity-Level Sentiment Analysis (ELSA): An exploratory task survey](https://mlsanity.com/paper/2304.14241/) (2023)

## Built on by (most cited first, 4 of 4)

- [Mixed Feelings: Cross-Domain Sentiment Classification of Patient Feedback](https://mlsanity.com/paper/2501.19134/) (2025)
- [HealthcareNLP: where are we and what is next?](https://mlsanity.com/paper/2512.08617/) (2025)
- [LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments](https://mlsanity.com/paper/2604.10834/) (2026)
- [Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model](https://mlsanity.com/paper/2605.03624/) (2026)

## Abstract (arXiv)

Sentiment analysis is an important tool for aggregating patient voices, in order to provide targeted improvements in healthcare services. A prerequisite for this is the availability of in-domain data annotated for sentiment. This article documents an effort to add sentiment annotations to free-text comments in patient surveys collected by the Norwegian Institute of Public Health (NIPH). However, annotation can be a time-consuming and resource-intensive process, particularly when it requires domain expertise. We therefore also evaluate a possible alternative to human annotation, using large language models (LLMs) as annotators. We perform an extensive evaluation of the approach for two openly available pretrained LLMs for Norwegian, experimenting with different configurations of prompts and in-context learning, comparing their performance to human annotators. We find that even for zero-shot runs, models perform well above the baseline for binary sentiment, but still cannot compete with human annotators on the full dataset.