mlsanityData to 7 Oct 2026

Contextual bandits with entropy-based human feedback

arXiv 2502.08759 ↗First posted 12 Feb 2025cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In recent years, preference-based human feedback mechanisms have become essential for enhancing model performance across diverse applications, including conversational AI systems such as ChatGPT. However, existing approaches often neglect critical aspects, such as model uncertainty and the variability in feedback quality. To address these challenges, we introduce an entropy-based human feedback framework for contextual bandits, which dynamically balances exploration and exploitation by soliciting expert feedback only when model entropy exceeds a predefined threshold. Our method is model-agnostic and can be seamlessly integrated with any contextual bandit agent employing stochastic policies. Through comprehensive experiments, we show that our approach achieves significant performance improvements while requiring minimal human feedback, even under conditions of suboptimal feedback quality. This work not only presents a novel strategy for feedback solicitation but also highlights the robustness and efficacy of incorporating human guidance into machine learning systems. Our code is publicly available: https://github.com/BorealisAI/CBHF

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 56 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

12 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback16 Apr
  2. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  3. 2023Adversarial Rewards in Universal Learning for Contextual Bandits14 Feb
  4. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback28 Oct

Other references · 8

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021EE-Net: Exploitation-Exploration Neural Networks in Contextual Banditscited by 43
  4. 2014Thompson sampling with the online bootstrapcited by 19
  5. 2023Borda Regret Minimization for Generalized Linear Dueling Banditscited by 17
  6. 2020FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedbackcited by 15
  7. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  8. 2024CAREForMe: Contextual Multi-Armed Bandit Recommendation Framework for Mental Healthcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Autoformalization of Agent Instructions into Policy-as-Codecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Kernelized Offline Contextual Dueling Bandits
  2. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  4. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
  5. 2025Active Human Feedback Collection via Neural Contextual Dueling Bandits
  6. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2025Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback