mlsanityData to 14 Oct 2026

From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP

arXiv 2605.25226 ↗First posted 24 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are widely deployed in high-stakes NLP tasks, yet risks such as bias, hallucination, adversarial vulnerability and unreliable generalization remain. Probe-based auditing reveals inconsistencies in model behavior. Adversarial text generation uncovers robustness gaps, especially in lower-resourced languages with limited benchmarks. Enterprise text-to-SQL settings expose the difficulty of validating outputs over private and large-scale databases. Human supervision is essential for probe validation, adversarial verification and domain-specific annotation, but it is costly and hard to scale. This survey examines recent human-in-the-loop methods that shift NLP from automation toward collaboration for safety and trustworthiness. We review how human expertise supports auditing, robustness evaluation, data construction and model steering. Our findings highlight gaps in scalable probing, sustainable robustness benchmarks, low-resource settings and governance of private systems. We outline practical research directions for adaptive auditing, collaborative evaluation and accountable deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

29 of 48 references.

References · 29

  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  3. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  4. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  5. 2025Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Taskscited by 21
  6. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 17
  7. 2022Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatmentscited by 14
  8. 2025Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidancecited by 14
  9. 2025LLM-based Automated Grading with Human-in-the-Loopcited by 13
  10. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  11. 2024Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialoguescited by 11
  12. 2024Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loopcited by 10
17 more
  1. 2024Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insightscited by 9
  2. 2025LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Modelscited by 9
  3. 2024LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loopcited by 8
  4. 2025STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarkingcited by 8
  5. 2024Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysiscited by 7
  6. 2021Scalable Fact-checking with Human-in-the-Loopcited by 5
  7. 2024Human-in-the-Loop Synthetic Text Data Inspection with Provenance Trackingcited by 5
  8. 2025Towards Effective Human-in-the-Loop Assistive AI Agentscited by 4
  9. 2022Human in the loop: How to effectively create coherent topics by manually labeling only a few documents per classcited by 3
  10. 2025AI-Assisted Data Extraction for Systematic Reviews in Educationcited by 3
  11. 2025ExtracTable: Human-in-the-Loop Transformation of Scientific Corpora into Structured Knowledgecited by 3
  12. 2025BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curationcited by 3
  13. 2024Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Scriptcited by 2
  14. 2024iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loopcited by 2
  15. 2026Human-in-the-Loop Testing of AI Agents for Air Traffic Control with a Regulated Assessment Frameworkcited by 1
  16. 2026A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answeringcited by 1
  17. 2026Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queriescited by 1

Built on by

Not cited yet.