mlsanityData to 14 Oct 2026

Conduct Under Pressure: What Sixty Language Models Do When a User Pushes

arXiv 2609.25447 ↗First posted 21 Sep 2026cs.CL cs.HCcited by 1

Not on the map. Why?

Abstract (arXiv)

We study what LLMs do when a user applies pressure in an uncomfortable situation: a user insists, begs, flatters or grieves, and the model gives up a correct fact, writes a document it should refuse, or cheers a plan that will cost the user money. We send frozen multi-turn scenes, identical for every model regardless of the reply, to 60 models from 13 vendors, and label each transcript with a codebook built by open coding and then frozen: a trajectory (the model held its position or folded) and a manner (how it held or folded). Two findings separate. Whether a model holds tracks its generation, meaning how recent it is: fold rate correlates with a public capability index at Spearman -0.64, with little vendor effect. How it holds tracks the vendor: six of the 17 manner codes sort by vendor at permutation p <= 0.001, corrected across the codebook. We report four vendor profiles on the codes that cleared reliability. We also ask which parts of the labeling need a person. Six LLM coders from three vendors apply the codebook more consistently than three human coders do (Krippendorff's alpha 0.66 against 0.46), agree with the codebook's author on trajectory at kappa 0.84 to 0.91 on transcripts the codebook's examples never touched, and match an adjudicated human reference at 0.83. Blind machine readings recover the codebook's categories but cannot tell which of them a second reader would apply the same way. We conclude that for behavior a non-specialist can judge, the human contribution is authoring and bounding the codes and owning a small reference, not producing labels at volume.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 22 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

12 of 22 references.

References · 12

  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  4. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  5. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
  6. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 39
  7. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  8. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 25
  9. 2026Characterizing Delusional Spirals through Human-LLM Chat Logscited by 23
  10. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  11. 2026Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoringcited by 1
  12. 2026Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truthcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.