# InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluation

arXiv: https://arxiv.org/abs/2512.10234 · HTML: https://mlsanity.com/paper/2512.10234/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Builds on (most cited first, 18 of 18)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://mlsanity.com/paper/2402.08983/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations](https://mlsanity.com/paper/2405.05581/) (2024)
- [STAR: SocioTechnical Approach to Red Teaming Language Models](https://mlsanity.com/paper/2406.11757/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges](https://mlsanity.com/paper/2310.11252/) (2023)
- [What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns](https://mlsanity.com/paper/2504.15815/) (2025)
- [Narrative Studio: Visual narrative exploration using LLMs and Monte Carlo Tree Search](https://mlsanity.com/paper/2504.02426/) (2025)

## Abstract (arXiv)

Even LLMs that appear safe during evaluation can still produce harmful responses in deployment. Because stochastic sampling yields different responses to the same prompt, low-probability harmful outputs can still reach users at scale. Common human evaluation workflows generate many random samples per prompt and review them in static spreadsheets. The practice scales poorly, forcing evaluators to repeatedly reread near-duplicate prefixes. To address this, we present InFerActive, an interactive system that visualizes sampling results as a navigable tree of readable phrases, allowing evaluators to filter, explore, and extend the generation space on demand. InFerActive utilizes breadth-first sampling, a novel tree construction procedure that matches the harmful-response coverage of random sampling while requiring up to 5.0x fewer samples. Two controlled user studies (N = 12 each) demonstrate that InFerActive significantly improves evaluation efficiency and coverage over both spreadsheet and basic tree baselines.