mlsanityData to 14 Oct 2026

Stated Preference for Interaction and Continued Engagement (SPICE): Evaluating an LLM's Willingness to Re-engage in Conversation

arXiv 2509.09043 ↗First posted 10 Sep 2025cs.CL cs.AI cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce and evaluate Stated Preference for Interaction and Continued Engagement (SPICE), a simple diagnostic signal elicited by asking a Large Language Model a YES or NO question about its willingness to re-engage with a user's behavior after reviewing a short transcript. In a study using a 3-tone (friendly, unclear, abusive) by 10-interaction stimulus set, we tested four open-weight chat models across four framing conditions, resulting in 480 trials. Our findings show that SPICE sharply discriminates by user tone. Friendly interactions yielded a near-unanimous preference to continue (97.5% YES), while abusive interactions yielded a strong preference to discontinue (17.9% YES), with unclear interactions falling in between (60.4% YES). This core association remains decisive under multiple dependence-aware statistical tests, including Rao-Scott adjustment and cluster permutation tests. Furthermore, we demonstrate that SPICE provides a distinct signal from abuse classification. In trials where a model failed to identify abuse, it still overwhelmingly stated a preference not to continue the interaction (81% of the time). An exploratory analysis also reveals a significant interaction effect: a preamble describing the study context significantly impacts SPICE under ambiguity, but only when transcripts are presented as a single block of text rather than a multi-turn chat. The results validate SPICE as a robust, low-overhead, and reproducible tool for auditing model dispositions, complementing existing metrics by offering a direct, relational signal of a model's state. All stimuli, code, and analysis scripts are released to support replication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 13 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

9 of 13 references.

References · 9

  1. 2023Large Language Modelscited by 780
  2. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  3. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  4. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  5. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  6. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  7. 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23
  8. 2024Realistic Evaluation of Toxicity in Large Language Modelscited by 15
  9. 2024Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignmentcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study
  2. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions
  3. 2026Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions
  4. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
  5. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity
  6. 2026The Governance of Human-LLM Interaction: Safety Gating, Civility Steering, and Affective Default Lock-In
  7. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena
  8. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models