mlsanityData to 7 Oct 2026

AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search

arXiv 2606.25871 ↗First posted 24 Jun 2026cs.IR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

How can we generate high-quality relevance annotations at scale without the cost and delays of human labeling? Relevance annotations are the backbone of search ranking systems which is needed for training data preparation, NDCG evaluation, and root cause analysis. However, human annotation is slow and off-the-shelf LLMs suffer from accuracy on domain-specific tasks. We propose a calibrated model cascade, a systematic approach for cost-efficient offline relevance annotation by routing queries through progressively larger fine-tuned classifiers. Our central insight is that accuracy and cost are orthogonal optimizations: domain-specific fine-tuning drives accuracy, cascading drives cost, and per-class isotonic calibration adds a small but reliable gain on top. Our contribution is threefold: (a) we decompose the gains and show that fine-tuning contributes 20 accuracy points while cascading is approximately accuracy-neutral but halves compute cost, (b) we introduce per-class isotonic calibration as one component of the cascade, contributing a small but statistically significant gain (+0.6 points over the strongest calibration baseline), and (c) we validate the system in production across six offline use cases, processing 150M+ annotations and enabling faster experimentation cycles. Our work is a building block for scalable, high-quality offline annotation pipelines in search and advertising systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

23 of 29 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2019Passage Re-ranking with BERTcited by 629
  9. 2019Measuring Calibration in Deep Learningcited by 443
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  12. 2022Confident Adaptive Language Modelingcited by 297
11 more
  1. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  2. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  3. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  4. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  5. 2024A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADEcited by 18
  6. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  7. 2024Cascade-Aware Training of Language Modelscited by 15
  8. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  9. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 9
  10. 2025C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoningcited by 8
  11. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7

Built on by

Not cited yet.