# Monotonicity as an Architectural Bias for Robust Language Models

arXiv: https://arxiv.org/abs/2602.02686 · HTML: https://mlsanity.com/paper/2602.02686/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM](https://mlsanity.com/paper/2309.14348/) (2023)
- [Deep Lattice Networks and Partial Monotonic Functions](https://mlsanity.com/paper/1709.06680/) (2017)
- [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025)
- [Mission Impossible: A Statistical Perspective on Jailbreaking LLMs](https://mlsanity.com/paper/2408.01420/) (2024)
- [Training Transformers with Enforced Lipschitz Constants](https://mlsanity.com/paper/2507.13338/) (2025)
- [Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations](https://mlsanity.com/paper/2505.02537/) (2025)
- [Adversarial Suffix Filtering: a Defense Pipeline for LLMs](https://mlsanity.com/paper/2505.09602/) (2025)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [Stability Analysis of Monotone Systems via Max-separable Lyapunov Functions](https://mlsanity.com/paper/1607.07966/) (2016)

## Abstract (arXiv)

Large language models (LLMs) are known to exhibit brittle behavior under adversarial prompts and jailbreak attacks, even after extensive alignment and fine-tuning. This fragility reflects a broader challenge of modern neural language models: small, carefully structured perturbations in high-dimensional input spaces can induce large and unpredictable changes in internal semantic representations and output. We investigate monotonicity as an architectural inductive bias for improving the robustness of Transformer-based language models. Monotonicity constrains semantic transformations so that strengthening information, evidence, or constraints cannot lead to regressions in the corresponding internal representations. Such order-preserving behavior has long been exploited in control and safety-critical systems to simplify reasoning and improve robustness, but has traditionally been viewed as incompatible with the expressivity required by neural language models. We show that this trade-off is not inherent. By enforcing monotonicity selectively in the feed-forward sublayers of sequence-to-sequence Transformers -- while leaving attention mechanisms unconstrained -- we obtain monotone language models that preserve the performance of their pretrained counterparts. This architectural separation allows negation, contradiction, and contextual interactions to be introduced explicitly through attention, while ensuring that subsequent semantic refinement is order-preserving. Empirically, monotonicity substantially improves robustness: adversarial attack success rates drop from approximately 69% to 19%, while standard summarization performance degrades only marginally.