mlsanityData to 7 Oct 2026

Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing

arXiv 2607.28814 ↗First posted 30 Jul 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In Motivational Interviewing (MI), a client's sustain talk (arguments for the status quo) calls for the counselor to roll with resistance, a move that can fail in two opposite ways: capitulation (abandoning the change agenda to preserve rapport) or confrontation (arguing or directing, overriding the client's autonomy). We introduce a two-axis evaluation of counselor responses, anchored in the Motivational Interviewing Treatment Integrity (MITI) code, Goal Persistence (GP) and Relational Attunement (RA), yielding a four-quadrant framing in which rolling with resistance is high on both, and we ask whether penalizing one failure through preference optimization teaches rolling with resistance or provokes its opposite. From the expert-annotated AnnoMI corpus we build topic-disjoint Direct Preference Optimization data whose preference sets differ only in which failure is rejected, using on-policy negatives. An automatic judge, validated against AnnoMI's expert labels and rechecked by trained human coders, scores blind pairwise win-rates against each base under a firewall in which disjoint model families generate, label, and judge. Across three aligned instruction models spanning the Qwen and Llama families, penalizing confrontation reliably lowers goal persistence below parity, on every base and in every seed run, a robust cost, whereas the attunement gain is base-dependent, present on two of the three bases but absent on the third. Penalizing capitulation is inert, because these models rarely capitulate on-policy, so the trade is gated by each base's failure profile. A prompt-only control raises attunement without the goal-persistence cost, locating the cost in the optimization rather than in attunement itself.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 27 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

15 of 27 references.

References · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  11. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  12. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
3 more
  1. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  2. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  3. 2024Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generationcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Compliance: A Resistance-Informed Motivation Reasoning Framework for Challenging Psychological Client Simulation
  2. 2026MIThinker: A Plug-and-Play Policy-Optimized Thinker For Motivational Interviewing Counseling
  3. 2025CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration
  4. 2026Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
  5. 2026CALM-IT: Generating Realistic Long-Form Motivational Interviewing Dialogues with Dual-Actor Conversational Dynamics Tracking
  6. 2025Preference Learning Unlocks LLMs' Psycho-Counseling Skills
  7. 2026When Do LLM Preferences Predict Downstream Behavior?
  8. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?