mlsanityData to 14 Oct 2026

Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guarantees

arXiv 2410.15761 ↗First posted 21 Oct 2024cs.CL cs.LG stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large Language Models excel in generative tasks but exhibit inefficiencies in structured text selection, particularly in extractive question answering. This challenge is magnified in resource-constrained environments, where deploying multiple specialized models for different tasks is impractical. We propose a Learning-to-Defer framework that allocates queries to specialized experts, ensuring high-confidence predictions while optimizing computational efficiency. Our approach integrates a principled allocation strategy with theoretical guarantees on optimal deferral that balances performance and cost. Empirical evaluations on SQuADv1, SQuADv2, and TriviaQA demonstrate that our method enhances answer reliability while significantly reducing computational overhead, making it well-suited for scalable and efficient EQA deployment.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 46 of its 107 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

46 of 107 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
34 more
  1. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  2. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  3. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  4. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  5. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 210
  6. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  7. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  9. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  10. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  11. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  12. 2024Exploiting LLM Quantizationcited by 65
  13. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  14. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  15. 2021Calibration and Consistency of Adversarial Surrogate Lossescited by 47
  16. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  17. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  18. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43
  19. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  20. 2024Faster Cascades via Speculative Decodingcited by 38
  21. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  22. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  23. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  24. 2024Regression with Multi-Expert Deferralcited by 29
  25. 2023Ranking with Abstentioncited by 27
  26. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  27. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  28. 2024Cardinality-Aware Set Prediction and Top-k Classificationcited by 23
  29. 2024Multi-Label Learning with Stronger Consistency Guaranteescited by 22
  30. 2024H-Consistency Guarantees for Regressioncited by 21
  31. 2024Enhanced H-Consistency Boundscited by 18
  32. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  33. 2022On Efficient Approximate Queries over Machine Learning Modelscited by 7
  34. 2018Structured Output Learning with Abstention: Application to Accurate Opinion Predictioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
  2. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  3. 2025Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Datacited by 15
  4. 2025Budgeted Multiple-Expert Deferralcited by 13
  5. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 7
  6. 2025Principled Algorithms for Optimizing Generalized Metrics in Binary Classificationcited by 6
  7. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 6
  8. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  9. 2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 3
  10. 2026Learning When to Remember: Risk-Sensitive Contextual Bandits for Abstention-Aware Memory Retrieval in LLM-Based Coding Agentscited by 3
  11. 2026Optimized Deferral for Imbalanced Settingscited by 2
  12. 2026Linear-Core Surrogates: Smooth Loss Functions with Linear Rates for Classification and Structured Predictioncited by 2
3 more
  1. 2026Online Learning-to-Defer with Varying Expertscited by 1
  2. 2026Strategic Decision Support for AI Agentscited by 1
  3. 2026Should I stay or should I show? Learning to selectively disclose informationcited by 0