mlsanityData to 14 Oct 2026

SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification

arXiv 2311.15983 ↗First posted 27 Nov 2023cs.LG cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Among the many tasks that Large Language Models (LLMs) have revolutionized is text classification. Current text classification paradigms, however, rely solely on the output of the final layer in the LLM, with the rich information contained in internal neurons largely untapped. In this study, we present SPIN: a model-agnostic framework that sparsifies and integrates internal neurons of intermediate layers of LLMs for text classification. Specifically, SPIN sparsifies internal neurons by linear probing-based salient neuron selection layer by layer, avoiding noise from unrelated neurons and ensuring efficiency. The cross-layer salient neurons are then integrated to serve as multi-layered features for the classification head. Extensive experimental results show our proposed SPIN significantly improves text classification accuracy, efficiency, and interpretability.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

33 of 50 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
21 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  6. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  11. 2021Dynamic Neural Networks: A Surveycited by 408
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  13. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  14. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  15. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  16. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  17. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  18. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  19. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  20. 2020Finding Experts in Transformer Modelscited by 28
  21. 2023Applying BERT and ChatGPT for Sentiment Analysis of Lyme Disease in Scientific Literaturecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 11
  2. 2025SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Modelscited by 6
  3. 2025Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMscited by 5
  4. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  5. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalizationcited by 2
  6. 2026MINER: Mining Multimodal Internal Representation for Efficient Retrievalcited by 2
  7. 2025Order from Chaos: Comparative Study of Ten Leading LLMs on Unstructured Data Categorizationcited by 1
  8. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  9. 2026Tandem Reinforcement Learning with Verifiable Rewardscited by 0
  10. 2026Agent Safety From Within: Detecting Harmful Trajectories from LLM Internal Statescited by 0