# Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

arXiv: https://arxiv.org/abs/2608.02455 · HTML: https://mlsanity.com/paper/2608.02455/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 97 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 30 of 34)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence](https://mlsanity.com/paper/2404.00029/) (2024)
- [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020)
- [CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms](https://mlsanity.com/paper/1905.11353/) (2019)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective](https://mlsanity.com/paper/2009.11469/) (2020)
- [Human Expertise in Algorithmic Prediction](https://mlsanity.com/paper/2402.00793/) (2024)
- [Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?](https://mlsanity.com/paper/2408.14141/) (2024)
- [Rank Aggregation via Heterogeneous Thurstone Preference Models](https://mlsanity.com/paper/1912.01211/) (2019)
- [Learning Inconsistent Preferences with Gaussian Processes](https://mlsanity.com/paper/2006.03847/) (2020)
- [ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding](https://mlsanity.com/paper/2303.13186/) (2023)
- [Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation](https://mlsanity.com/paper/1810.08851/) (2018)
- [A System-Level Analysis of Conference Peer Review](https://mlsanity.com/paper/2303.09020/) (2023)
- [Explaining Preferences with Shapley Values](https://mlsanity.com/paper/2205.13662/) (2022)
- [RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Services](https://mlsanity.com/paper/2507.10605/) (2025)
- [MalLight: Influence-Aware Coordinated Traffic Signal Control for Traffic Signal Malfunctions](https://mlsanity.com/paper/2408.09768/) (2024)
- [Any to Full: Prompting Depth Anything for Depth Completion in One Stage](https://mlsanity.com/paper/2603.05711/) (2026)
- [FAP-CD: Fairness-Driven Age-Friendly Community Planning via Conditional Diffusion Generation](https://mlsanity.com/paper/2412.16699/) (2024)
- [False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking](https://mlsanity.com/paper/1605.05860/) (2016)
- [A Cost-Effective Framework for Preference Elicitation and Aggregation](https://mlsanity.com/paper/1805.05287/) (2018)
- [Fast and Robust Rank Aggregation against Model Misspecification](https://mlsanity.com/paper/1905.12341/) (2019)
- [CureGraph: Contrastive Multi-Modal Graph Representation Learning for Urban Living Circle Health Profiling and Prediction](https://mlsanity.com/paper/2501.07157/) (2025)
- [HypeMed: Enhancing Medication Recommendations with Hypergraph-Based Patient Relationships](https://mlsanity.com/paper/2603.18459/) (2026)
- [SemanticCAP: Chromatin Accessibility Prediction Enhanced by Features Learning from a Language Model](https://mlsanity.com/paper/2204.02130/) (2022)
- [Design and Evaluation of Crowd-sourcing Platforms Based on Users Confidence Judgments](https://mlsanity.com/paper/2212.05777/) (2022)
- [Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction](https://mlsanity.com/paper/2410.05550/) (2024)
- [AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data](https://mlsanity.com/paper/2411.13584/) (2024)
- [Towards 3D Objectness Learning in an Open World](https://mlsanity.com/paper/2510.17686/) (2025)
- [Spatio-Temporal Hierarchical Causal Models](https://mlsanity.com/paper/2511.20558/) (2025)

## Abstract (arXiv)

Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model's scores by an isotonic projection onto the order, enforcing ordinal consistency while preserving as much of the model's quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.