Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning

arXiv 2510.09887 ↗First posted 10 Oct 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Vision and language models frequently ignore semantically critical input edits, defaulting to pretraining priors. For example, models will confidently assert a five-legged dog has four legs; consequently, on the VLMBias benchmark, GPT 5.2 and Claude Sonnet 4.6 achieve only 4.6% and 0% accuracy, respectively. Existing methods address this problem through building up datasets that covers the underrepresented inputs to tune the policy function π(y mid x), where x and y refer to input prompts and responses, respectively. However, prompting baselines yield gains of under 3% on VLMBias due to the low probability density of rare prompts. To bypass this bottleneck, we propose abductive preference learning to optimize the abductive policy π(x mid y). We prove this amplifies forward policy improvements by a factor of q(y)/p(x), where p(·) and q(·) denote the marginal probabilities of the prompt and response, yielding the largest gains on the rarest prompts. Furthermore, we demonstrate that for translation invariant pairwise preference learning methods, such as DPO, estimating π(x mid y) reduces to a structural data swap that compares prompts for a fixed response, requiring no architectural changes. Empirically, abductive preference learning delivers large gains on long-tail prompt sensitivity: on VLMBias, A-DPO raises accuracy from 3% to 44% (14×), outperforming GPT-5.2 (4.6%) and all closed-source VLMs except Gemini~3~Flash; on Inverse-IFEval, Multi-DPOP reaches 65--84%, surpassing GPT-5 (73.7%) at the 9B scale while preserving IFBench, unlike DPO which degrades it by 8--12%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 28 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

22 of 28 references.

Closest ancestors · 4

Same direction, newest first

Other references · 18

2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
2018On the Spectral Bias of Neural Networkscited by 1,321
2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
2023Inverse Scaling: When Bigger Isn't Bettercited by 205

6 more

2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way