Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
Not on the map. Why?
Abstract (arXiv)
Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call behavioral relapse, or revocation inertia. No existing instrument measures this influence per clause, predicts it before delivery, or repairs it under matched budgets. sysname closes the three gaps through the model API alone: a contract ledger pairs every constraint with an executable checker, records revocations as tombstones, and compiles the net constraint state ahead of time into a single specification; a sequential ablation probe measures per-clause adherence and incremental behavioral effect; a repair ladder operates under token- and attempt-matched budgets. On dataname (NTasks HumanEval tasks, NClauses verified checkers), relapse at an 8B operating point climbs from ScaleDelayedMTwo to ScaleDelayedMEight as constraint load grows, while stronger models sit at floor. Under matched checkers, model, and budget, ahead-of-time compilation significantly reduces relapse against a no-ledger verifier-retry baseline (RestoreDiff, 95% CI RestoreDiffCI, p RestoreDiffP); adaptive ladder interventions stacked on top add no detectable gain (95% confidence excludes gains ≥ LadderExcludedGain). The probe predicts relapse before delivery (AUROC AurocPrimary); a one-sentence tombstone note recovers about a third of the compilation effect and survives a placebo control. At CostDeliveryFactor delivery overhead and CostTotalHedged of API compute for every result, revocation failure becomes a measurable, predictable, and repairable property of dialogue state rather than an invisible one.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 12 Aug 2026.
Builds on
30 of 40 references.
References · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2018Datasheets for Datasetscited by 1,386
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2023Teaching Large Language Models to Self-Debugcited by 988
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
18 more
- 2025Kimi K2: Open Agentic Intelligencecited by 567
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2023Efficient Guided Generation for Large Language Modelscited by 200
- 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
- 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
- 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
- 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
- 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
- 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
- 2021Preregistering NLP Researchcited by 25
- 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 19
- 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
- 2025Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attackscited by 6
- 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6
- 2024Estimating Causal Effects of Text Interventions Leveraging LLMscited by 4
- 2026Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideationcited by 3
- 2025ContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generationcited by 2
- 2022The Role of Placebo Samples in Observational Studiescited by 1
Built on by
Not cited yet.