# Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy

arXiv: https://arxiv.org/abs/2205.11966 · HTML: https://mlsanity.com/paper/2205.11966/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [From Arguments to Key Points: Towards Automatic Argument Summarization](https://mlsanity.com/paper/2005.01619/) (2020)
- [Quantitative Argument Summarization and Beyond: Cross-Domain Key Point Analysis](https://mlsanity.com/paper/2010.05369/) (2020)
- [HINT3: Raising the bar for Intent Detection in the Wild](https://mlsanity.com/paper/2009.13833/) (2020)
- [Every Bite Is an Experience: Key Point Analysis of Business Reviews](https://mlsanity.com/paper/2106.06758/) (2021)
- [Expressive Interviewing: A Conversational System for Coping with COVID-19](https://mlsanity.com/paper/2007.03819/) (2020)
- [Project Debater APIs: Decomposing the AI Grand Challenge](https://mlsanity.com/paper/2110.01029/) (2021)
- [Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems](https://mlsanity.com/paper/2204.05158/) (2022)

## Built on by (most cited first, 4 of 4)

- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours](https://mlsanity.com/paper/2208.01483/) (2022)
- [Understanding Enthymemes in Argument Maps: Bridging Argument Mining and Logic-based Argumentation](https://mlsanity.com/paper/2408.08648/) (2024)
- [IntentGrasp: A Comprehensive Benchmark for Intent Understanding](https://mlsanity.com/paper/2605.06832/) (2026)

## Abstract (arXiv)

The COVID-19 pandemic has made a huge global impact and cost millions of lives. As COVID-19 vaccines were rolled out, they were quickly met with widespread hesitancy. To address the concerns of hesitant people, we launched VIRA, a public dialogue system aimed at addressing questions and concerns surrounding the COVID-19 vaccines. Here, we release VIRADialogs, a dataset of over 8k dialogues conducted by actual users with VIRA, providing a unique real-world conversational dataset. In light of rapid changes in users' intents, due to updates in guidelines or in response to new information, we highlight the important task of intent discovery in this use-case. We introduce a novel automatic evaluation framework for intent discovery, leveraging the existing intent classifier of VIRA. We use this framework to report baseline intent discovery results over VIRADialogs, that highlight the difficulty of this task.