mlsanityData to 7 Oct 2026

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models

arXiv 2402.14688 ↗First posted 22 Feb 2024cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present an approach called Q-probing to adapt a pre-trained language model to maximize a task-specific reward function. At a high level, Q-probing sits between heavier approaches such as finetuning and lighter approaches such as few shot prompting, but can also be combined with either. The idea is to learn a simple linear function on a model's embedding space that can be used to reweight candidate completions. We theoretically show that this sampling procedure is equivalent to a KL-constrained maximization of the Q-probe as the number of samples increases. To train the Q-probes we consider either reward modeling or a class of novel direct policy learning objectives based on importance weighted policy gradients. With this technique, we see gains in domains with ground-truth rewards (code generation) as well as implicit rewards defined by preference data, even outperforming finetuning in data-limited regimes. Moreover, a Q-probe can be trained on top of an API since it only assumes access to sampling and embeddings. Code: https://github.com/likenneth/q_probe .

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 58 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

48 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2015Trust Region Policy Optimizationcited by 2,503
31 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  13. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  14. 2022PAL: Program-aided Language Modelscited by 635
  15. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  16. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  17. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  18. 2023Llemma: An Open Language Model For Mathematicscited by 385
  19. 2022Self-critiquing models for assisting human evaluatorscited by 372
  20. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  21. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  22. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  23. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  24. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  25. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  26. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  27. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  28. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  29. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  30. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  31. 2023Learning to Generate Better Than Your LLMcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  2. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  3. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  4. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  5. 2024Truthful Aggregation of LLMs with an Application to Online Advertisingcited by 29
  6. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  7. 2025Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RLcited by 11
  8. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  9. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Spacecited by 8
  10. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  11. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  12. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1
3 more
  1. 2025Memory-Driven Self-Improvement for Decision Making with Large Language Modelscited by 0
  2. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  3. 2026Inverse RL Helps Align AI by Imitating Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Discriminative Policy Optimization for Token-Level Reward Models
  2. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  3. 2025ToolRL: Reward is All Tool Learning Needs
  4. 2025RewardAnything: Generalizable Principle-Following Reward Models
  5. 2024T-REG: Preference Optimization with Token-Level Reward Regularization
  6. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
  7. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
  8. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective