# Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop

arXiv: https://arxiv.org/abs/2211.01354 · HTML: https://mlsanity.com/paper/2211.01354/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization](https://mlsanity.com/paper/2011.01421/) (2020)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition](https://mlsanity.com/paper/2208.14536/) (2022)
- [ner and pos when nothing is capitalized](https://mlsanity.com/paper/1903.11222/) (2019)
- [BLINK with Elasticsearch for Efficient Entity Linking in Business Conversations](https://mlsanity.com/paper/2205.04438/) (2022)
- [Improving Punctuation Restoration for Speech Transcripts via External Data](https://mlsanity.com/paper/2110.00560/) (2021)
- [An Effective, Performant Named Entity Recognition System for Noisy Business Telephone Conversation Transcripts](https://mlsanity.com/paper/2209.13736/) (2022)
- [Utilizing Bidirectional Encoder Representations from Transformers for Answer Selection](https://mlsanity.com/paper/2011.07208/) (2020)
- [Robustness to Capitalization Errors in Named Entity Recognition](https://mlsanity.com/paper/1911.05241/) (2019)
- [Entity-level Sentiment Analysis in Contact Center Telephone Conversations](https://mlsanity.com/paper/2210.13401/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries](https://mlsanity.com/paper/2304.13620/) (2023)
- [Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers](https://mlsanity.com/paper/2306.04504/) (2023)
- [AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching](https://mlsanity.com/paper/2305.17619/) (2023)
- [AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents](https://mlsanity.com/paper/2510.08149/) (2025)
- [LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deployment](https://mlsanity.com/paper/2503.19090/) (2025)

## Abstract (arXiv)

Telephone transcription data can be very noisy due to speech recognition errors, disfluencies, etc. Not only that annotating such data is very challenging for the annotators, but also such data may have lots of annotation errors even after the annotation job is completed, resulting in a very poor model performance. In this paper, we present an active learning framework that leverages human in the loop learning to identify data samples from the annotated dataset for re-annotation that are more likely to contain annotation errors. In this way, we largely reduce the need for data re-annotation for the whole dataset. We conduct extensive experiments with our proposed approach for Named Entity Recognition and observe that by re-annotating only about 6% training instances out of the whole dataset, the F1 score for a certain entity type can be significantly improved by about 25%.