mlsanityData to 14 Oct 2026

Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions

arXiv 2601.09724 ↗First posted 27 Dec 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models exhibit systematic negation sensitivity, yet no operational framework exists to measure this vulnerability at deployment scale, especially in high-stakes decisions. We introduce Syntactic Framing Fragility (SFF), a framework for quantifying decision consistency under logically equivalent syntactic transformations. SFF isolates syntactic effects via Logical Polarity Normalization, enabling direct comparison across positive and negative framings while controlling for polarity inversion, and provides the Syntactic Variation Index (SVI) as a robustness metric suitable for CI/CD integration. Auditing 23 models across 14 high-stakes scenarios (39,975 decisions), we establish ground-truth effect sizes for a phenomenon previously characterized only qualitatively and find that open-source models exhibit 2.2x higher fragility than commercial counterparts. Negation-bearing syntax is the dominant failure mode, with some models endorsing actions at 80-97% rates even when asked whether agents not act. These patterns are consistent with negation suppression failure documented in prior work, with chain-of-thought reasoning reducing fragility in some but not all cases. We provide scenario-stratified risk profiles and offer an operational checklist compatible with EU AI Act and NIST RMF requirements. Code, data, and scenarios will be released upon publication.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 27 Dec 2025.

Builds on

27 of 45 references.

References · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2020Aligning AI With Shared Human Valuescited by 723
  7. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  9. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  10. 2022Using cognitive psychology to understand GPT-3cited by 398
  11. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  12. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
15 more
  1. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  2. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  3. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  4. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  5. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  6. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  7. 2024Efficient multi-prompt evaluation of LLMscited by 73
  8. 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 44
  9. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 40
  10. 2024Quantifying perturbation impacts for large language modelscited by 8
  11. 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 8
  12. 2025Statistical Hypothesis Testing for Auditing Robustness in Language Modelscited by 8
  13. 2024Informed AI Regulation: Comparing the Ethical Frameworks of Leading LLM Chatbots Using an Ethics-Based Audit to Assess Moral Reasoning and Normative Valuescited by 5
  14. 2025Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learningcited by 2
  15. 2025Polarity-Aware Probing for Quantifying Latent Alignment in Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?cited by 12
  2. 2026Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Fine-Tuning Language Models for Ethical Ambiguity: A Comparative Study of Alignment with Human Responses
  2. 2025Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs
  3. 2026Mood Matters: How Syntactic Sensitivity Undermines Safety Alignment
  4. 2026When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
  5. 2026CONDESION-BENCH: Conditional Decision-Making of Large Language Models in Compositional Action Space
  6. 2025Deontological Keyword Bias: The Impact of Modal Expressions on Normative Judgments of Language Models
  7. 2024Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
  8. 2026Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability