mlsanityData to 14 Oct 2026

Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities

arXiv 2505.21191 ↗First posted 27 May 2025cs.CL cs.LGcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The finetuning of Large Language Models (LLMs) has significantly advanced their instruction-following capabilities, yet the underlying computational mechanisms driving these improvements remain poorly understood. This study systematically examines how fine-tuning reconfigures LLM computations by isolating and analyzing instruction-specific sparse components, i.e., neurons in dense models and both neurons and experts in Mixture-of-Experts (MoE) architectures. In particular, we introduce HexaInst, a carefully curated and balanced instructional dataset spanning six distinct categories, and propose SPARCOM, a novel analytical framework comprising three key contributions: (1) a method for identifying these sparse components, (2) an evaluation of their functional generality and uniqueness, and (3) a systematic comparison of their alterations. Through experiments, we demonstrate functional generality, uniqueness, and the critical role of these components in instruction execution. By elucidating the relationship between fine-tuning-induced adaptations and sparse computational substrates, this work provides deeper insights into how LLMs internalize instruction-following behavior for the trustworthy LLM community.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 37 of its 49 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

37 of 49 references.

References · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
25 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  4. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  5. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  6. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  7. 2024How do Large Language Models Handle Multilingualism?cited by 150
  8. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  9. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  11. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  12. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  13. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  14. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 92
  15. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  16. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  17. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  18. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
  19. 2025LLMs for Explainable AI: A Comprehensive Surveycited by 45
  20. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  21. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  22. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  23. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  24. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  25. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  2. 2026A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Modelcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  2. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models
  3. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  4. 2024Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
  5. 2023Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
  6. 2024Do LLMs "know" internally when they follow instructions?
  7. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  8. 2026Patches of Nonlinearity: Instruction Vectors in Large Language Models