Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?
Not on the map. Why?
Abstract (arXiv)
The output of a Language Model can be tampered with while the model is writing it. A simple test can thus be constructed by evaluating the model's perception of this external perturbation. In this spirit, a simple benchmark is built in which a single word is consistently substituted with another in the generation process. We call this method Sleight of Word. Two distinct axes are measured: metrics that relate to the model's surprise, as well as an evaluation of the textual reaction for 19 different open-weight language models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 30 Aug 2026.
Builds on
8 of 14 references.
References · 8
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
Built on by
Not cited yet.