# Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learning

arXiv: https://arxiv.org/abs/2011.04883 · HTML: https://mlsanity.com/paper/2011.04883/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 5 of 5)

- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments](https://mlsanity.com/paper/2104.08560/) (2021)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media](https://mlsanity.com/paper/2210.06331/) (2022)
- [Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions](https://mlsanity.com/paper/2507.15692/) (2025)

## Abstract (arXiv)

Datasets extracted from social networks and online forums are often prone to the pitfalls of natural language, namely the presence of unstructured and noisy data. In this work, we seek to enable the collection of high-quality question-answer datasets from social media by proposing a novel task for automated quality analysis and data cleaning: question-answer (QA) plausibility. Given a machine or user-generated question and a crowd-sourced response from a social media user, we determine if the question and response are valid; if so, we identify the answer within the free-form response. We design BERT-based models to perform the QA plausibility task, and we evaluate the ability of our models to generate a clean, usable question-answer dataset. Our highest-performing approach consists of a single-task model which determines the plausibility of the question, followed by a multi-task model which evaluates the plausibility of the response as well as extracts answers (Question Plausibility AUROC=0.75, Response Plausibility AUROC=0.78, Answer Extraction F1=0.665).