mlsanityData to 14 Oct 2026

Can LLMs Follow Simple Rules?

arXiv 2311.04235 ↗First posted 6 Nov 2023cs.AI cs.CL cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) are deployed with increasing real-world responsibilities, it is important to be able to specify and constrain the behavior of these systems in a reliable manner. Model developers may wish to set explicit rules for the model, such as "do not generate abusive content", but these may be circumvented by jailbreaking techniques. Existing evaluations of adversarial attacks and defenses on LLMs generally require either expensive manual review or unreliable heuristic checks. To address this issue, we propose Rule-following Language Evaluation Scenarios (RuLES), a programmatic framework for measuring rule-following ability in LLMs. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. Each scenario has a programmatic evaluation function to determine whether the model has broken any rules in a conversation. Our evaluations of proprietary and open models show that almost all current models struggle to follow scenario rules, even on straightforward test cases. We also demonstrate that simple optimization attacks suffice to significantly increase failure rates on test cases. We conclude by exploring two potential avenues for improvement: test-time steering and supervised fine-tuning.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 52 of its 77 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

52 of 77 references.

References · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2022Red Teaming Language Models with Language Modelscited by 1,026
  13. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  14. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  15. 2020Aligning AI With Shared Human Valuescited by 723
  16. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  17. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  19. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  20. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  21. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  22. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  23. 2023Are aligned neural networks adversarially aligned?cited by 315
  24. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  25. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  26. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  27. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  28. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  29. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  30. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 175
  31. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  32. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  33. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  34. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  35. 2023Large Language Models can Learn Rulescited by 69
  36. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
  37. 2023Tractable Control for Autoregressive Language Generationcited by 65
  38. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 17; 2026: 9.

46 papers.

Sort:

By area:

  1. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  2. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  3. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  4. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  5. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  6. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  7. 2025A Closer Look at System Prompt Robustnesscited by 32
  8. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  9. 2024SysBench: Can Large Language Models Follow System Messages?cited by 28
  10. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  11. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 27
  12. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 26
34 more
  1. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 26
  2. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  3. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 24
  4. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 24
  5. 2024Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Modelscited by 23
  6. 2024SoFA: Shielded On-the-fly Alignment via Priority Rule Followingcited by 18
  7. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  8. 2025On Verifiable Legal Reasoning: A Multi-Agent Framework with Formalized Knowledge Representationscited by 18
  9. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  10. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  11. 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agentscited by 14
  12. 2025Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approachcited by 14
  13. 2025In-Context Watermarks for Large Language Modelscited by 10
  14. 2025LLMs for Customized Marketing Content Generation and Evaluation at Scalecited by 9
  15. 2024Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?cited by 8
  16. 2024Human-AI Safety: A Descendant of Generative AI and Control Systems Safetycited by 7
  17. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  18. 2024Baichuan Alignment Technical Reportcited by 6
  19. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  20. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5
  21. 2025ReflecSched: Solving Dynamic Flexible Job-Shop Scheduling via LLM-Powered Hierarchical Reflectioncited by 5
  22. 2025Security Steerability is All You Needcited by 4
  23. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  24. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?cited by 3
  25. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 3
  26. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  27. 2026Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integrationcited by 1
  28. 2026Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policiescited by 1
  29. 2026AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governancecited by 0
  30. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMscited by 0
  31. 2026Engineering Trustworthy Agentic AI for Critical Systemscited by 0
  32. 2026Learning to Follow In-Context Watermark Instructions via Self-Distillationcited by 0
  33. 2026Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agentscited by 0
  34. 2026Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Examscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
  2. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
  3. 2024Coercing LLMs to do and reveal (almost) anything
  4. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
  5. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  6. 2025Learned-Rule-Augmented Large Language Model Evaluators
  7. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  8. 2025Attack and defense techniques in large language models: A survey and new perspectives