# From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP

arXiv: https://arxiv.org/abs/2605.25226 · HTML: https://mlsanity.com/paper/2605.25226/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 29 of 29)

- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks](https://mlsanity.com/paper/2507.15821/) (2025)
- [Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI](https://mlsanity.com/paper/2409.09467/) (2024)
- [Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments](https://mlsanity.com/paper/2212.09683/) (2022)
- [Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance](https://mlsanity.com/paper/2507.17131/) (2025)
- [LLM-based Automated Grading with Human-in-the-Loop](https://mlsanity.com/paper/2504.05239/) (2025)
- [Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks](https://mlsanity.com/paper/2011.02511/) (2020)
- [Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues](https://mlsanity.com/paper/2412.09049/) (2024)
- [Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop](https://mlsanity.com/paper/2407.05925/) (2024)
- [Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights](https://mlsanity.com/paper/2404.04281/) (2024)
- [LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Models](https://mlsanity.com/paper/2504.00752/) (2025)
- [LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop](https://mlsanity.com/paper/2402.09346/) (2024)
- [STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking](https://mlsanity.com/paper/2507.03674/) (2025)
- [Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis](https://mlsanity.com/paper/2405.03677/) (2024)
- [Scalable Fact-checking with Human-in-the-Loop](https://mlsanity.com/paper/2109.10992/) (2021)
- [Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking](https://mlsanity.com/paper/2404.18881/) (2024)
- [Towards Effective Human-in-the-Loop Assistive AI Agents](https://mlsanity.com/paper/2507.18374/) (2025)
- [Human in the loop: How to effectively create coherent topics by manually labeling only a few documents per class](https://mlsanity.com/paper/2212.09422/) (2022)
- [AI-Assisted Data Extraction for Systematic Reviews in Education](https://mlsanity.com/paper/2501.11840/) (2025)
- [ExtracTable: Human-in-the-Loop Transformation of Scientific Corpora into Structured Knowledge](https://mlsanity.com/paper/2506.03221/) (2025)
- [BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation](https://mlsanity.com/paper/2510.13853/) (2025)
- [Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script](https://mlsanity.com/paper/2412.12478/) (2024)
- [iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loop](https://mlsanity.com/paper/2412.12644/) (2024)
- [Human-in-the-Loop Testing of AI Agents for Air Traffic Control with a Regulated Assessment Framework](https://mlsanity.com/paper/2601.04288/) (2026)
- [A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering](https://mlsanity.com/paper/2602.05512/) (2026)
- [Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queries](https://mlsanity.com/paper/2602.15738/) (2026)

## Abstract (arXiv)

Large language models are widely deployed in high-stakes NLP tasks, yet risks such as bias, hallucination, adversarial vulnerability and unreliable generalization remain. Probe-based auditing reveals inconsistencies in model behavior. Adversarial text generation uncovers robustness gaps, especially in lower-resourced languages with limited benchmarks. Enterprise text-to-SQL settings expose the difficulty of validating outputs over private and large-scale databases. Human supervision is essential for probe validation, adversarial verification and domain-specific annotation, but it is costly and hard to scale. This survey examines recent human-in-the-loop methods that shift NLP from automation toward collaboration for safety and trustworthiness. We review how human expertise supports auditing, robustness evaluation, data construction and model steering. Our findings highlight gaps in scalable probing, sustainable robustness benchmarks, low-resource settings and governance of private systems. We outline practical research directions for adaptive auditing, collaborative evaluation and accountable deployment.