# On Multiplicative Integration with Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1606.06630 · HTML: https://mlsanity.com/paper/1606.06630/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2016.

## Built on by (most cited first, 15 of 15)

- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [High-Order Pooling for Graph Neural Networks with Tensor Decomposition](https://mlsanity.com/paper/2205.11691/) (2022)
- [Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units](https://mlsanity.com/paper/2104.02899/) (2021)
- [Explainable Natural Language Processing with Matrix Product States](https://mlsanity.com/paper/2112.08628/) (2021)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [Language Modeling Using Tensor Trains](https://mlsanity.com/paper/2405.04590/) (2024)
- [Revisiting Bi-Linear State Transitions in Recurrent Neural Networks](https://mlsanity.com/paper/2505.21749/) (2025)
- [Spiking Manifesto](https://mlsanity.com/paper/2512.11843/) (2025)
- [Stability Analysis of Various Symbolic Rule Extraction Methods from Recurrent Neural Network](https://mlsanity.com/paper/2402.02627/) (2024)
- [A Tensor Decomposition Perspective on Second-order RNNs](https://mlsanity.com/paper/2406.05045/) (2024)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)
- [Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2605.16318/) (2026)
- [Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication](https://mlsanity.com/paper/2601.19915/) (2026)
- [On the Role of Depth in the Expressivity of RNNs](https://mlsanity.com/paper/2604.02201/) (2026)
- [Gating Enables Curvature: A Geometric Expressivity Gap in Attention](https://mlsanity.com/paper/2604.14702/) (2026)

## Abstract (arXiv)

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.