# Online Active Model Selection for Pre-trained Classifiers

arXiv: https://arxiv.org/abs/2010.09818 · HTML: https://mlsanity.com/paper/2010.09818/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits](https://mlsanity.com/paper/2403.07213/) (2024)
- [All models are wrong, some are useful: Model Selection with Limited Labels](https://mlsanity.com/paper/2410.13609/) (2024)
- [Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning](https://mlsanity.com/paper/2410.21547/) (2024)
- [Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection](https://mlsanity.com/paper/2504.13938/) (2025)
- [Consensus-Driven Active Model Selection](https://mlsanity.com/paper/2507.23771/) (2025)
- [Anytime Model Selection in Linear Bandits](https://mlsanity.com/paper/2307.12897/) (2023)
- [On the Necessity of Collaboration for Online Model Selection with Decentralized Data](https://mlsanity.com/paper/2404.09494/) (2024)
- [Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations](https://mlsanity.com/paper/2108.06514/) (2021)
- [Contextual Active Model Selection](https://mlsanity.com/paper/2207.06030/) (2022)
- [Contextual Decision Trees](https://mlsanity.com/paper/2207.06355/) (2022)
- [Bayesian Online Learning for Consensus Prediction](https://mlsanity.com/paper/2312.07679/) (2023)
- [Budgeted Online Model Selection and Fine-Tuning via Federated Learning](https://mlsanity.com/paper/2401.10478/) (2024)
- [Large Language Model Selection with Limited Annotations](https://mlsanity.com/paper/2605.24981/) (2026)
- [Online Learning with LLM Experts from Limited Feedback](https://mlsanity.com/paper/2609.05820/) (2026)
- [Large Language Model Selection with Limited Annotations](https://mlsanity.com/paper/2510.09418/) (2025)
- [Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection](https://mlsanity.com/paper/2609.04388/) (2026)
- [Which LLM to pick? Online Active Model Selection for Large Language Models](https://mlsanity.com/paper/2610.01592/) (2026)
- [Bayesian Learning to Select Fair Classifiers](https://mlsanity.com/paper/2610.20514/) (2026)

## Abstract (arXiv)

Given k pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.