mlsanityData to 14 Oct 2026

Can Input Attributions Explain Inductive Reasoning in In-Context Learning?

arXiv 2412.15628 ↗First posted 20 Dec 2024cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Interpreting the internal process of neural models has long been a challenge. This challenge remains relevant in the era of large language models (LLMs) and in-context learning (ICL); for example, ICL poses a new issue of interpreting which example in the few-shot examples contributed to identifying/solving the task. To this end, in this paper, we design synthetic diagnostic tasks of inductive reasoning, inspired by the generalization tests typically adopted in psycholinguistics. Here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates it. The question is whether conventional input attribution (IA) methods can track such a reasoning process, i.e., identify the influential example, in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 67 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

16 of 67 references.

References · 16

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Mistral 7Bcited by 3,164
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
  6. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  7. 2024Scaling and evaluating sparse autoencoderscited by 553
  8. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  10. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  11. 2016Understanding Neural Networks through Representation Erasurecited by 224
  12. 2015A Neural Conversational Modelcited by 196
4 more
  1. 2015Visualizing and Understanding Recurrent Networkscited by 160
  2. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  3. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 67
  4. 2014Extraction of Salient Sentences from Labelled Documentscited by 50

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainabilitycited by 4
  2. 2025Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
  2. 2024On the Role of Model Prior in Real-World Inductive Reasoning
  3. 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
  4. 2025Is In-Context Learning Learning?
  5. 2023Hypothesis Search: Inductive Reasoning with Language Models
  6. 2023Concept-aware Training Improves In-context Learning Ability of Language Models
  7. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
  8. 2024An Incomplete Loop: Deductive, Inductive, and Abductive Reasoning in Language Models