# Cost-Aware Post-Hoc Deferral Under Calibration and Shift: An Environmental AI Case Study

arXiv: https://arxiv.org/abs/2609.09235 · HTML: https://mlsanity.com/paper/2609.09235/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles](https://mlsanity.com/paper/2001.11086/) (2020)
- [How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks](https://mlsanity.com/paper/2305.15508/) (2023)
- [Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints](https://mlsanity.com/paper/2403.06906/) (2024)
- [Exploring Multimodal Foundation AI and Expert-in-the-Loop for Sustainable Management of Wild Salmon Fisheries in Indigenous Rivers](https://mlsanity.com/paper/2505.06637/) (2025)
- [Density-Ratio Losses for Post-Hoc Learning to Defer](https://mlsanity.com/paper/2605.19557/) (2026)

## Abstract (arXiv)

Choosing a deferral policy for a frozen classifier requires more than ranking uncertain cases: confidence may be miscalibrated, errors have unequal costs, reviewers can err, and deployment data can leave calibration support. We study these interactions through EcoTrust, a post-hoc framework that compares automatic action with review using a six-group error-risk estimator, class-asymmetric costs, reviewer accuracy, and an optional support gate. On a Columbia River thermal-stress testbed, the learned estimator improves error-ranking area under the receiver operating characteristic curve from 0.869 to 0.889, but Chow's confidence rule has lower in-distribution cost (0.416 versus 0.567 per day). Across 12 off-the-shelf backends, learned risk and a calibration-matched, class-aware confidence estimator each beat raw Chow on six; a paired year-block bootstrap does not resolve their mean cost difference. In transfer to ten river stations, the gate flags every case and becomes an always-review fallback, attaining the lowest cost on eight stations only when review is perfect and unconstrained. These results characterize decision boundaries on one controlled task: richer risk signals do not reliably improve on calibrated confidence, and detected extrapolation does not imply transferable case-level ranking.