# KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

arXiv: https://arxiv.org/abs/2505.17919 · HTML: https://mlsanity.com/paper/2505.17919/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://mlsanity.com/paper/2310.06786/) (2023)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias](https://mlsanity.com/paper/2110.13905/) (2021)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [Gradient Descent Quantizes ReLU Network Features](https://mlsanity.com/paper/1803.08367/) (2018)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [MegaMath: Pushing the Limits of Open Math Corpora](https://mlsanity.com/paper/2504.02807/) (2025)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization](https://mlsanity.com/paper/2307.12851/) (2023)

## Abstract (arXiv)

Residual connections are foundational to modern neural networks, yet the standard additive update provides no explicit mechanism for structured inter-channel interaction. We introduce KITINet (KInetics Theory Inspired Network), a training-time module with no additional trainable parameters that augments residual updates with stochastic pairwise feature interactions inspired by collision sampling in kinetic theory. KITINet reshapes channel groups as particles and mixes their features according to relative distance and velocity, while leaving the underlying residual layer trainable in the usual way. The module is disabled at inference, so the deployed model is identical to the original backbone and incurs no additional inference cost. Across language pre-training and fine-tuning, image classification, and PDE operator learning, KITINet demonstrates broad and robust performance gains across diverse tasks and architectures. We further observe enhanced parameter condensation across several settings and develop a simplified stochastic analysis that provides a mechanistic account of how collision-inspired interactions can promote condensation dynamics.