# HumanAL: Calibrating Human Matching Beyond a Single Task

arXiv: https://arxiv.org/abs/2205.03209 · HTML: https://mlsanity.com/paper/2205.03209/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 6 May 2022.

## Builds on (most cited first, 18 of 18)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Text Matching as Image Recognition](https://mlsanity.com/paper/1602.06359/) (2016)
- [FIND: Human-in-the-Loop Debugging Deep Text Classifiers](https://mlsanity.com/paper/2010.04987/) (2020)
- [Low-resource Deep Entity Resolution with Transfer and Active Learning](https://mlsanity.com/paper/1906.08042/) (2019)
- [Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark](https://mlsanity.com/paper/1905.10425/) (2019)
- [Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification](https://mlsanity.com/paper/1906.06678/) (2019)
- [ZeroER: Entity Resolution using Zero Labeled Examples](https://mlsanity.com/paper/1908.06049/) (2019)
- [MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matching](https://mlsanity.com/paper/1905.10289/) (2019)
- [A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching](https://mlsanity.com/paper/2003.13114/) (2020)
- [Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions](https://mlsanity.com/paper/1801.02546/) (2018)
- [Deep Indexed Active Learning for Matching Heterogeneous Entity Representations](https://mlsanity.com/paper/2104.03986/) (2021)
- [Deep Transfer Learning for Multi-source Entity Linkage via Domain Adaptation](https://mlsanity.com/paper/2110.14509/) (2021)
- [Reducing Uncertainty of Schema Matching via Crowdsourcing with Accuracy Rates](https://mlsanity.com/paper/1809.04017/) (2018)
- [Learning to Characterize Matching Experts](https://mlsanity.com/paper/2012.01229/) (2020)
- [PoWareMatch: a Quality-aware Deep Learning Approach to Improve Human Schema Matching](https://mlsanity.com/paper/2109.07321/) (2021)
- [Optimizing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management](https://mlsanity.com/paper/1610.05377/) (2016)

## Built on by (most cited first, 2 of 2)

- [The Battleship Approach to the Low Resource Entity Matching Problem](https://mlsanity.com/paper/2311.15685/) (2023)
- [Humans, Machine Learning, and Language Models in Union: A Cognitive Study on Table Unionability](https://mlsanity.com/paper/2506.12990/) (2025)

## Abstract (arXiv)

This work offers a novel view on the use of human input as labels, acknowledging that humans may err. We build a behavioral profile for human annotators which is used as a feature representation of the provided input. We show that by utilizing black-box machine learning, we can take into account human behavior and calibrate their input to improve the labeling quality. To support our claims and provide a proof-of-concept, we experiment with three different matching tasks, namely, schema matching, entity matching and text matching. Our empirical evaluation suggests that the method can improve the quality of gathered labels in multiple settings including cross-domain (across different matching tasks).