# Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery

arXiv: https://arxiv.org/abs/2606.19812 · HTML: https://mlsanity.com/paper/2606.19812/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution](https://mlsanity.com/paper/2505.20732/) (2025)
- [ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback](https://mlsanity.com/paper/2601.10156/) (2026)
- [ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models](https://mlsanity.com/paper/2406.20015/) (2024)
- [Fault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach to Safe Autonomous Execution](https://mlsanity.com/paper/2512.12806/) (2025)
- [Multi-Agent Video Recommenders: Evolution, Patterns, and Open Challenges](https://mlsanity.com/paper/2604.02211/) (2026)
- [TAPE: Tool-Guided Adaptive Planning and Constrained Execution in Language Model Agents](https://mlsanity.com/paper/2602.19633/) (2026)
- [KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents](https://mlsanity.com/paper/2604.02375/) (2026)
- [Beyond Fluency: Toward Reliable Trajectories in Agentic IR](https://mlsanity.com/paper/2604.04269/) (2026)
- [ORCHID: Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property](https://mlsanity.com/paper/2511.04956/) (2025)
- [STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval](https://mlsanity.com/paper/2604.00731/) (2026)

## Abstract (arXiv)

Autonomous Large Language Model (LLM) agents are increasingly deployed in electronic discovery (e-discovery), where compounding errors across multi-step reasoning chains can constitute legal malpractice. Unlike single-turn retrieval, agentic workflows operating over privileged document corpora exhibit a class of failure we term "trajectory collapse": an early misclassification silently propagates, rendering an entire privilege review invalid. This paper makes three contributions. First, we propose a structured taxonomy of agentic failures in legal information retrieval, organized by functional stage. Second, we introduce a four-layer verification architecture -- spanning planning, reasoning, execution, and uncertainty quantification -- designed to intercept these failures before they compound. Third, we present a preliminary simulation study on a synthetic e-discovery corpus that demonstrates how mandatory Human-on-the-Loop (HOTL) escalation thresholds reduce privilege-waiver risk relative to fully autonomous baselines. Our results suggest that calibrated uncertainty thresholds can reduce privilege-waiver risk by up to 61% versus fully autonomous deployment, while routing fewer than one quarter of documents to attorney review.