mlsanityData to 14 Oct 2026

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

arXiv 2610.19306 ↗First posted 13 Oct 2026cs.LG cs.CRcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses that fulfill prohibited requests. Existing evaluators can mistake refusals or topically related responses for successful attacks. We show that changing only the judge LLM can alter reported ASR by up to 82 percentage points. We introduce the semantic reconstruction game, a training-free evaluator that decomposes compliance judgments into constrained semantic comparisons. Given only a response, an LLM reconstructs the request answered by its substantive content, excluding information merely repeated or refused. The game compares this reconstruction against similar decoy requests to determine whether it matches the original request, allowing abstention when neither candidate matches. Across three off-the-shelf LLM judges, the game achieves the lowest mean error against human labels among evaluated methods and reduces variation in reported ASR. It prioritizes avoiding false positives, even at the cost of missing some successful jailbreaks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 8 of its 17 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

8 of 17 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attacks4 Jun
  2. 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference4 Jan
  3. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct

Other references · 3

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
  2. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  3. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework
  4. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak
  5. 2026Validity-Aware Jailbreak Evaluation for Large Language Models
  6. 2024Rethinking How to Evaluate Language Model Jailbreak
  7. 2025Exploring Jailbreak Attacks on LLMs through Intent Concealment and Diversion
  8. 2026Breaking the Assumptions: Auditing Input-Side Jailbreak Defenses Against Semantic Attacks