# Classification Under Human Assistance

arXiv: https://arxiv.org/abs/2006.11845 · HTML: https://mlsanity.com/paper/2006.11845/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2020.

## Built on by (most cited first, 30 of 42)

- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [Improving Expert Predictions with Conformal Prediction](https://mlsanity.com/paper/2201.12006/) (2022)
- [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022)
- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Designing Decision Support Systems Using Counterfactual Prediction Sets](https://mlsanity.com/paper/2306.03928/) (2023)
- [In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer](https://mlsanity.com/paper/2311.01106/) (2023)
- [Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer](https://mlsanity.com/paper/2407.13732/) (2024)
- [Learning to Defer with Limited Expert Predictions](https://mlsanity.com/paper/2304.07306/) (2023)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer](https://mlsanity.com/paper/2506.20650/) (2025)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Training Data Subset Selection for Regression with Controlled Generalization Error](https://mlsanity.com/paper/2106.12491/) (2021)
- [Learning to Make Adherence-Aware Advice](https://mlsanity.com/paper/2310.00817/) (2023)
- [Human Expertise in Algorithmic Prediction](https://mlsanity.com/paper/2402.00793/) (2024)
- [A Causal Framework for Evaluating Deferring Systems](https://mlsanity.com/paper/2405.18902/) (2024)
- [Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks](https://mlsanity.com/paper/2409.12255/) (2024)
- [Budgeted Multiple-Expert Deferral](https://mlsanity.com/paper/2510.26706/) (2025)
- [Improving Learning-to-Defer Algorithms Through Fine-Tuning](https://mlsanity.com/paper/2112.10768/) (2021)
- [Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets](https://mlsanity.com/paper/2406.06671/) (2024)
- [Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers](https://mlsanity.com/paper/2204.01334/) (2022)
- [AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems](https://mlsanity.com/paper/2402.06287/) (2024)
- [Towards Human-AI Complementarity in Matching Tasks](https://mlsanity.com/paper/2508.13285/) (2025)
- [Neural Estimation of Submodular Functions with Applications to Differentiable Subset Selection](https://mlsanity.com/paper/2210.11033/) (2022)
- [A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems](https://mlsanity.com/paper/2407.12710/) (2024)
- [Learning To Guide Human Decision Makers With Vision-Language Models](https://mlsanity.com/paper/2403.16501/) (2024)
- [Collaborative Intelligence in Sequential Experiments: A Human-in-the-Loop Framework for Drug Discovery](https://mlsanity.com/paper/2405.03942/) (2024)
- [Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Framework](https://mlsanity.com/paper/2410.08783/) (2024)
- [Narrowing Action Choices with AI Improves Human Sequential Decisions](https://mlsanity.com/paper/2510.16097/) (2025)
- [When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer](https://mlsanity.com/paper/2602.17144/) (2026)

## Abstract (arXiv)

Most supervised learning models are trained for full automation. However, their predictions are sometimes worse than those by human experts on some specific instances. Motivated by this empirical observation, our goal is to design classifiers that are optimized to operate under different automation levels. More specifically, we focus on convex margin-based classifiers and first show that the problem is NP-hard. Then, we further show that, for support vector machines, the corresponding objective function can be expressed as the difference of two functions f = g - c, where g is monotone, non-negative and γ-weakly submodular, and c is non-negative and modular. This representation allows a recently introduced deterministic greedy algorithm, as well as a more efficient randomized variant of the algorithm, to enjoy approximation guarantees at solving the problem. Experiments on synthetic and real-world data from several applications in medical diagnosis illustrate our theoretical findings and demonstrate that, under human assistance, supervised learning models trained to operate under different automation levels can outperform those trained for full automation as well as humans operating alone.