# Interactive Prompt Debugging with Sequence Salience

arXiv: https://arxiv.org/abs/2404.07498 · HTML: https://mlsanity.com/paper/2404.07498/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI](https://mlsanity.com/paper/1902.01876/) (2019)
- [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018)
- [Extraction of Salient Sentences from Labelled Documents](https://mlsanity.com/paper/1412.6815/) (2014)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [Using Captum to Explain Generative Language Models](https://mlsanity.com/paper/2312.05491/) (2023)
- [LMdiff: A Visual Diff Tool to Compare Language Models](https://mlsanity.com/paper/2111.01582/) (2021)
- [LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations](https://mlsanity.com/paper/2401.12576/) (2024)
- [The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets](https://mlsanity.com/paper/2009.11023/) (2020)

## Built on by (most cited first, 6 of 6)

- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [TracLLM: A Generic Framework for Attributing Long Context LLMs](https://mlsanity.com/paper/2506.04202/) (2025)
- [DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviors](https://mlsanity.com/paper/2602.05446/) (2026)
- [Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models](https://mlsanity.com/paper/2505.12509/) (2025)
- [Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics](https://mlsanity.com/paper/2603.05832/) (2026)
- [PromptFlow: Training Prompts Like Neural Networks](https://mlsanity.com/paper/2510.12246/) (2025)

## Abstract (arXiv)

We present Sequence Salience, a visual tool for interactive prompt debugging with input salience methods. Sequence Salience builds on widely used salience methods for text classification and single-token prediction, and extends this to a system tailored for debugging complex LLM prompts. Our system is well-suited for long texts, and expands on previous work by 1) providing controllable aggregation of token-level salience to the word, sentence, or paragraph level, making salience over long inputs tractable; and 2) supporting rapid iteration where practitioners can act on salience results, refine prompts, and run salience on the new output. We include case studies showing how Sequence Salience can help practitioners work with several complex prompting strategies, including few-shot, chain-of-thought, and constitutional principles. Sequence Salience is built on the Learning Interpretability Tool, an open-source platform for ML model visualizations, and code, notebooks, and tutorials are available at http://goo.gle/sequence-salience.