mlsanityData to 7 Oct 2026

COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier

arXiv 2510.01178 ↗First posted 1 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Selecting an optimal set of exemplars is critical for good performance of in-context learning. However, prior exemplar search methods narrowly optimize for predictive accuracy, critically neglecting model calibration--a key determinant of trustworthiness and safe deployment. In this paper, we formulate exemplar selection as a multi-objective optimization problem, explicitly targeting both the maximization of predictive accuracy and the minimization of expected calibration error. We solve this problem with a sample-efficient Combinatorial Bayesian Optimization algorithm (COM-BOM) to find the Pareto front that optimally trades off the two objectives of accuracy and calibration. We evaluate COM-BOM on multiple tasks from unsaturated MMLU-Pro benchmark and find that COM-BOM beats or matches the baselines at jointly optimizing the two objectives, while requiring a minimal number of LLM API calls.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

21 of 36 references.

References · 21

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  3. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  4. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  5. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  6. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  7. 2023Compositional Exemplars for In-context Learningcited by 143
  8. 2023Large Language Models in Law: A Surveycited by 128
  9. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 120
  10. 2021Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvementcited by 115
  11. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 109
  12. 2021Multi-Objective Bayesian Optimization over High-Dimensional Search Spacescited by 87
9 more
  1. 2020Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraintscited by 81
  2. 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
  3. 2021Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spacescited by 51
  4. 2020Mercer Features for Efficient Combinatorial Bayesian Optimizationcited by 27
  5. 2023A Study on the Calibration of In-context Learningcited by 21
  6. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  7. 2023Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimizationcited by 15
  8. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12
  9. 2023InstructEval: Systematic Evaluation of Instruction Selection Methodscited by 12

Built on by

Not cited yet.