mlsanityData to 14 Oct 2026

Trustworthy Human Computation: A Survey

arXiv 2210.12324 ↗First posted 22 Oct 2022cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Human computation is an approach to solving problems that prove difficult using AI only, and involves the cooperation of many humans. Because human computation requires close engagement with both "human populations as users" and "human populations as driving forces," establishing mutual trust between AI and humans is an important issue to further the development of human computation. This survey lays the groundwork for the realization of trustworthy human computation. First, the trustworthiness of human computation as computing systems, that is, trust offered by humans to AI, is examined using the RAS (Reliability, Availability, and Serviceability) analogy, which define measures of trustworthiness in conventional computer systems. Next, the social trustworthiness provided by human computation systems to users or participants is discussed from the perspective of AI ethics, including fairness, privacy, and transparency. Then, we consider human--AI collaboration based on two-way trust, in which humans and AI build mutual trust and accomplish difficult tasks through reciprocal collaboration. Finally, future challenges and research directions for realizing trustworthy human computation are discussed.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 211 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

34 of 211 references.

References · 34

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  5. 2018Datasheets for Datasetscited by 1,386
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  7. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  8. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  9. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  10. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  11. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  12. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
22 more
  1. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  2. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 139
  3. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  4. 2020Learning to Complement Humanscited by 124
  5. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  6. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  7. 2017Deep learning from crowdscited by 105
  8. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  9. 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 72
  10. 2021Hybrid Intelligencecited by 55
  11. 2020Does Explainable Artificial Intelligence Improve Human Decision-Making?cited by 49
  12. 2021The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systemscited by 39
  13. 2020Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracycited by 20
  14. 2018Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actionscited by 16
  15. 2014Cost-Effective HITs for Relative Similarity Comparisonscited by 14
  16. 2021Toward Annotator Group Bias in Crowdsourcingcited by 14
  17. 2021Crowdsourcing Evaluation of Saliency-based XAI Methodscited by 7
  18. 2018Algorithms for the Greater Good! On Mental Modeling and Acceptable Symbiosis in Human-AI Collaborationcited by 5
  19. 2015Quizz: Targeted crowdsourcing with a billion (potential) userscited by 4
  20. 2016FROG: A Fast and Reliable Crowdsourcing Framework (Technical Report)cited by 2
  21. 2016Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platformscited by 2
  22. 2019Beyond monetary incentives: experiments in paid microtask contests modelled as continuous-time markov chainscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024On the Fairness, Diversity and Reliability of Text-to-Image Generative Modelscited by 2