mlsanityData to 7 Oct 2026

Neural Activation Patterns Across Language Model Architectures: A Comprehensive Analysis of Cognitive Task Performance

arXiv 2605.15436 ↗First posted 14 May 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents a comprehensive analysis of neural activation patterns across six distinct large language model (LLM) architectures, examining their performance on twelve cognitive task categories. Through systematic measurement of final activation values, attention entropy, and sparsity patterns, we reveal fundamental differences in how encoder and decoder architectures process diverse cognitive tasks. Our analysis of 144 task-model combinations demonstrates that mathematical reasoning consistently produces the highest attention entropy across all architectures, while decoder models exhibit significantly higher sparsity patterns compared to encoder models. The findings provide critical insights into the computational characteristics of modern language models and their task-specific neural behaviors, with implications for model selection and optimization in big data applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

39 of 57 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Mistral 7Bcited by 3,065
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
27 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2023Textbooks Are All You Needcited by 561
  16. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  17. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  18. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  19. 2019Revealing the Dark Secrets of BERTcited by 300
  20. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  21. 2019On Identifiability in Transformerscited by 115
  22. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  23. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  24. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  25. 2015A Primer on Neural Network Models for Natural Language Processingcited by 75
  26. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  27. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning
  2. 2026Understanding Large Language Models
  3. 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
  4. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
  5. 2025Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
  6. 2026Thinking vs. NoThinking: Towards Interpreting Reasoning Mechanisms of Large Language Models via Sparse Autoencoders
  7. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis
  8. 2026Beyond representational alignment with brain-guided language models for robust reasoning