# A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries

arXiv: https://arxiv.org/abs/2303.06944 · HTML: https://mlsanity.com/paper/2303.06944/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Few-NERD: A Few-Shot Named Entity Recognition Dataset](https://mlsanity.com/paper/2105.07464/) (2021)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [Current Challenges and Visions in Music Recommender Systems Research](https://mlsanity.com/paper/1710.03208/) (2017)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark](https://mlsanity.com/paper/1905.10425/) (2019)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition](https://mlsanity.com/paper/2004.07493/) (2020)
- [Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019](https://mlsanity.com/paper/2005.05738/) (2020)
- [Flow Moods: Recommending Music by Moods on Deezer](https://mlsanity.com/paper/2207.11229/) (2022)
- [Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition](https://mlsanity.com/paper/2108.11857/) (2021)
- [An End-to-End Solution for Named Entity Recognition in eCommerce Search](https://mlsanity.com/paper/2012.07553/) (2020)
- [Noise Robust Named Entity Understanding for Voice Assistants](https://mlsanity.com/paper/2005.14408/) (2020)
- [Recognizing Musical Entities in User-generated Content](https://mlsanity.com/paper/1904.00648/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection](https://mlsanity.com/paper/2412.11851/) (2024)
- [Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation](https://mlsanity.com/paper/2511.16478/) (2025)
- [HyCoRec: Hypergraph-Enhanced Multi-Preference Learning for Alleviating Matthew Effect in Conversational Recommendation](https://mlsanity.com/paper/2607.17461/) (2026)
- [Mitigating Matthew Effect: Multi-Hypergraph Boosted Multi-Interest Self-Supervised Learning for Conversational Recommendation](https://mlsanity.com/paper/2607.18609/) (2026)
- [Beyond Musical Descriptors: Extracting Preference-Bearing Intent in Music Queries](https://mlsanity.com/paper/2602.12301/) (2026)
- [FacetCRS: Multi-Faceted Preference Learning for Pricking Filter Bubbles in Conversational Recommender System](https://mlsanity.com/paper/2609.20175/) (2026)

## Abstract (arXiv)

We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.