# Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi

arXiv: https://arxiv.org/abs/2501.12900 · HTML: https://mlsanity.com/paper/2501.12900/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Phase transitions in Restricted Boltzmann Machines with generic priors](https://mlsanity.com/paper/1612.03132/) (2016)
- [Towards a universal mechanism for successful deep learning](https://mlsanity.com/paper/2309.07537/) (2023)
- [Brain experiments imply adaptation mechanisms which outperform common AI learning algorithms](https://mlsanity.com/paper/2005.04106/) (2020)
- [Learning on tree architectures outperforms a convolutional feedforward network](https://mlsanity.com/paper/2211.11378/) (2022)
- [The mechanism underlying successful deep learning](https://mlsanity.com/paper/2305.18078/) (2023)
- [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024)

## Built on by (most cited first, 6 of 6)

- [Low-latency vision transformers via large-scale multi-head attention](https://mlsanity.com/paper/2506.23832/) (2025)
- [Tiny language models](https://mlsanity.com/paper/2507.14871/) (2025)
- [Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning](https://mlsanity.com/paper/2509.03407/) (2025)
- [Translation Entropy: A Statistical Framework for Evaluating Translation Systems](https://mlsanity.com/paper/2511.13180/) (2025)
- [Self-attention vector output similarities reveal how machines pay attention](https://mlsanity.com/paper/2512.21956/) (2025)
- [Single-Nodal Spontaneous Symmetry Breaking in NLP Models](https://mlsanity.com/paper/2601.20582/) (2026)

## Abstract (arXiv)

Convolutional neural networks (CNNs) evaluate short-range correlations in input images which progress along the layers, whereas vision transformer (ViT) architectures evaluate long-range correlations, using repeated transformer encoders composed of fully connected layers. Both are designed to solve complex classification tasks but from different perspectives. This study demonstrates that CNNs and ViT architectures stem from a unified underlying learning mechanism, which quantitatively measures the single-nodal performance (SNP) of each node in feedforward (FF) and multi-head attention (MHA) sub-blocks. Each node identifies small clusters of possible output labels, with additional noise represented as labels outside these clusters. These features are progressively sharpened along the transformer encoders, enhancing the signal-to-noise ratio. This unified underlying learning mechanism leads to two main findings. First, it enables an efficient applied nodal diagonal connection (ANDC) pruning technique without affecting the accuracy. Second, based on the SNP, spontaneous symmetry breaking occurs among the MHA heads, such that each head focuses its attention on a subset of labels through cooperation among its SNPs. Consequently, each head becomes an expert in recognizing its designated labels, representing a quantitative MHA modus vivendi mechanism. This statistical mechanics inspired viewpoint enables to reveal macroscopic behavior of the entire network from the microscopic performance of each node. These results are based on a compact convolutional transformer architecture trained on the CIFAR-100 and Flowers-102 datasets and call for their extension to other architectures and applications, such as natural language processing.