mlsanityData to 14 Oct 2026

Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis

arXiv 2603.05510 ↗First posted 18 Jan 2026cs.HC cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Developing and deploying AI applications in organizations is challenging when human decision authority and oversight are underspecified across the system lifecycle. Although Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged, operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented. We report a multi-source qualitative study: a retrospective diary study of a customer-support chatbot and semi-structured interviews with eight AI experts from academia and industry. Through five-cycle thematic analysis of 1,435 codewords, we derive four themes: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination. These themes provide empirical inputs for subsequent HITL framework design and validation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 18 Jan 2026.

Builds on

18 of 39 references.

References · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  5. 2021Ethical and social risks of harm from Language Modelscited by 966
  6. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  7. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  8. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  9. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  10. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  11. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  12. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 29
6 more
  1. 2016Human-In-The-Loop Person Re-Identificationcited by 19
  2. 2022Efficient, Uncertainty-based Moderation of Neural Networks Text Classifierscited by 7
  3. 2023Design Patterns for Machine Learning Based Systems with Human-in-the-Loopcited by 4
  4. 2025AI and Human Oversight: A Risk-Based Framework for Alignmentcited by 4
  5. 2023An HCAI Methodological Framework (HCAI-MF): Putting It Into Action to Enable Human-Centered AIcited by 3
  6. 2025Human in the Latent Loop (HILL): Interactively Guiding Model Training Through Human Intuitioncited by 2

Built on by

Not cited yet.