mlsanityData to 7 Oct 2026

The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Making

arXiv 2601.21439 ↗First posted 29 Jan 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) are widely documented to be sensitive to minor prompt perturbations and prone to sycophantic alignment, their robustness in consequential, rule-bound decision-making remains under-explored. We uncover a striking "Paradox of Robustness": despite their known lexical brittleness, aligned LLMs exhibit strong robustness to emotional framing effects in rule-bound institutional decision-making. Using a controlled perturbation framework across three high-stakes domains (healthcare, finance, and education), we find a negligible effect size (Cohen's h = 0.003) compared to the substantial biases observed in analogous human contexts (h in [0.3, 0.8]), approximately two orders of magnitude smaller. This invariance persists across eight models with diverse training paradigms, suggesting the mechanisms driving sycophancy and prompt sensitivity do not translate to failures in logical constraint satisfaction. While LLMs may be "brittle" to how a query is formatted, they appear considerably more stable against affective attempts to bias rule-bound decisions. To probe the boundary of this finding, we add two reviewer-driven side studies. A five-scenario immigration extension yields a small but statistically detectable +0.8 percentage point shift that remains within a pre-specified +/-3 percentage point Region of Practical Equivalence (ROPE), while a screening-level adversarial narrative pilot finds no meaningful decision shift under stronger LLM-generated prompts. We release a core benchmark (9 base scenarios x 18 condition variants = 162 unique prompts), code, and data to facilitate replicable evaluation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

33 of 54 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2023Towards Understanding Sycophancy in Language Modelscited by 988
21 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  9. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  10. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  11. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  13. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  14. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  15. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  16. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  17. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  18. 2023Turning large language models into cognitive modelscited by 66
  19. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  20. 2023Calibrating LLM-Based Evaluatorcited by 53
  21. 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.