BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluation
On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation
Abstract (arXiv)
Accurate evaluation of user satisfaction is critical for iterative development of conversational AI. However, for open-ended assistants, traditional A/B testing lacks reliable metrics: explicit feedback is sparse, while implicit metrics are ambiguous. To bridge this gap, we introduce BoRP (Bootstrapped Regression Probing), a scalable framework for high-fidelity satisfaction evaluation. Unlike generative approaches, BoRP leverages the geometric properties of LLM latent space. It employs a polarization-index-based bootstrapping mechanism to automate rubric generation and utilizes Partial Least Squares (PLS) to map hidden states to continuous scores. Experiments on industrial datasets show that BoRP (Qwen3-8B/14B) significantly outperforms generative baselines (even Qwen3-Max) in alignment with human judgments. Furthermore, BoRP reduces inference costs by orders of magnitude, enabling full-scale monitoring and highly sensitive A/B testing via CUPED.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 19 of its 41 references that are in the map. First posted on arXiv 26 Jan 2026.
Builds on
19 of 41 references.
References · 19
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
- 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
7 more
- 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
- 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
- 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 37
- 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
- 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
- 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 25
- 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue
- 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
- 2026EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
- 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
- 2026PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators
- 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences
- 2025Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning
- 2026CAFE: A Compound-AI Factorial Evaluation Framework