mlsanityData to 7 Oct 2026

Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

arXiv 2606.08090 ↗First posted 6 Jun 2026cs.DB cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluating a natural-language yes/no predicate over a document corpus under an accuracy target - the semantic filter - is a cornerstone of LLM-based data processing. Calling the LLM on every document (the oracle) is prohibitive, so cascades pair the oracle with a fast proxy. As deployed today, they leave four limitations on the table. (1) Each cascade family - model-free clustering, prebuilt small-LLM proxies, online-trained proxies - commits to a single representation and pipeline, and wins on only a narrow query regime. (2) The strongest online proxy invests in a custom training scheme on a bi-encoder over dense embeddings, missing the token-level evidence richer predicates require. (3) The proxy is trained against binary yes/no labels, wasting the LLM's per-document confidence at the boundary documents it most needs to learn. (4) Existing calibrations add a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost. We address these by (1) composing families adaptively - model-free clustering first, online proxy only when needed, with oracle calls shared across phases; (2) replacing the cosine bi-encoder with a hybrid of off-the-shelf token-aware models; (3) training the proxy with the oracle's per-document confidence as a soft label; and (4) a calibration that adds the safety margin only where the labeled sample is sparse. We are also the first to use the oracle's per-document confidence for three purposes: a query-level difficulty compass, a lower bound on the minimum oracle calls any proxy-based cascade can make, and the proxy's soft training label. At a 90% accuracy target on three 10K-document corpora, our methods are 1.6-2.0x faster than the best prior method per corpus and meet the target on 95% of queries; the BER-derived lower bound indicates a further ~4-20x of headroom for future work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 55 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

28 of 55 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2021Convex optimizationcited by 2,479
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
16 more
  1. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  2. 2017Selective Classification for Deep Neural Networkscited by 799
  3. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  4. 2019Passage Re-ranking with BERTcited by 629
  5. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  6. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  7. 2020Overview of the TREC 2019 deep learning trackcited by 282
  8. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  9. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  10. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  11. 2022Adaptive Conformal Predictions for Time Seriescited by 155
  12. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  13. 2017Cascade Ranking for Operational E-commerce Searchcited by 29
  14. 2020Approximate Selection with Guarantees using Proxiescited by 13
  15. 2025Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guaranteescited by 8
  16. 2025ScaleDoc: Scaling LLM-based Predicates over Large Document Collectionscited by 2

Built on by

Not cited yet.