mlsanityData to 7 Oct 2026

JoPA:Explaining Large Language Model's Generation via Joint Prompt Attribution

arXiv 2405.20404 ↗First posted 30 May 2024cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performances in complex text generation tasks. However, the contribution of the input prompt to the generated content still remains obscure to humans, underscoring the necessity of understanding the causality between input and output pairs. Existing works for providing prompt-specific explanation often confine model output to be classification or next-word prediction. Few initial attempts aiming to explain the entire language generation often treat input prompt texts independently, ignoring their combinatorial effects on the follow-up generation. In this study, we introduce a counterfactual explanation framework based on Joint Prompt Attribution, JoPA, which aims to explain how a few prompt texts collaboratively influences the LLM's complete generation. Particularly, we formulate the task of prompt attribution for generation interpretation as a combinatorial optimization problem, and introduce a probabilistic algorithm to search for the casual input combination in the discrete space. We define and utilize multiple metrics to evaluate the produced explanations, demonstrating both the faithfulness and efficiency of our framework.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

35 of 51 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  12. 2022MaPLe: Multi-modal Prompt Learningcited by 685
23 more
  1. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  7. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  8. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  9. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  10. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  11. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  12. 2020How does this interaction affect me? Interpretable attribution for feature interactionscited by 73
  13. 2022The Creativity of Text-to-Image Generationcited by 73
  14. 2020Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same Endcited by 66
  15. 2020Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifierscited by 35
  16. 2023Using Captum to Explain Generative Language Modelscited by 33
  17. 2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
  18. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  19. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inferencecited by 27
  20. 2023ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphscited by 27
  21. 2021Rationales for Sequential Predictionscited by 26
  22. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  23. 2019Cognitive Functions of the Brain: Perception, Attention and Memorycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Training Language Models to Explain Their Own Computationscited by 27
  2. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  3. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  4. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  5. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2
  6. 2026In-Context Credit Assignment via the Corecited by 1