# Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units arXiv: https://arxiv.org/abs/2104.02899 ยท HTML: https://mlsanity.com/paper/2104.02899/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 52 references that are in the map. - First posted on arXiv 7 Apr 2021. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014) - [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014) - [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019) - [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015) - [Theoretical Limitations of Self-Attention in Neural Sequence Models](https://mlsanity.com/paper/1906.06755/) (2019) - [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019) - [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019) - [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://mlsanity.com/paper/1506.07285/) (2015) - [Neural Programmer-Interpreters](https://mlsanity.com/paper/1511.06279/) (2015) - [Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets](https://mlsanity.com/paper/1503.01007/) (2015) - [Learning to Transduce with Unbounded Memory](https://mlsanity.com/paper/1506.02516/) (2015) - [LSTM Networks Can Perform Dynamic Counting](https://mlsanity.com/paper/1906.03648/) (2019) - [Making Neural Programming Architectures Generalize via Recursion](https://mlsanity.com/paper/1704.06611/) (2017) - [Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages](https://mlsanity.com/paper/1911.03329/) (2019) - [Multiplicative LSTM for sequence modelling](https://mlsanity.com/paper/1609.07959/) (2016) - [Learning Continuous Semantic Representations of Symbolic Expressions](https://mlsanity.com/paper/1611.01423/) (2016) - [Can Neural Networks Understand Logical Entailment?](https://mlsanity.com/paper/1802.08535/) (2018) - [On Multiplicative Integration with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06630/) (2016) - [Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning](https://mlsanity.com/paper/1807.01406/) (2018) - [The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations](https://mlsanity.com/paper/1711.05738/) (2017) - [Learning to Discover Efficient Mathematical Identities](https://mlsanity.com/paper/1406.1584/) (2014) - [The Neural State Pushdown Automata](https://mlsanity.com/paper/1909.05233/) (2019) - [Tree Memory Networks for Modelling Long-term Temporal Dependencies](https://mlsanity.com/paper/1703.04706/) (2017) - [Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack](https://mlsanity.com/paper/2004.07623/) (2020) - [Can recursive neural tensor networks learn logical reasoning?](https://mlsanity.com/paper/1312.6192/) (2013) - [Graph Memory Networks for Molecular Activity Prediction](https://mlsanity.com/paper/1801.02622/) (2018) - [Compositional Generalization with Tree Stack Memory Units](https://mlsanity.com/paper/1911.01545/) (2019) ## Built on by (most cited first, 8 of 8) - [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023) - [On the Computational Complexity and Formal Hierarchy of Second Order Recurrent Neural Networks](https://mlsanity.com/paper/2309.14691/) (2023) - [Investigating Symbolic Capabilities of Large Language Models](https://mlsanity.com/paper/2405.13209/) (2024) - [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023) - [Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules](https://mlsanity.com/paper/2109.10476/) (2021) - [A Flawed Dataset for Symbolic Equation Verification](https://mlsanity.com/paper/2105.11479/) (2021) - [RLEEGNet: Integrating Brain-Computer Interfaces with Adaptive AI for Intuitive Responsiveness and High-Accuracy Motor Imagery Classification](https://mlsanity.com/paper/2402.09465/) (2024) - [Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights](https://mlsanity.com/paper/2410.03154/) (2024) ## Abstract (arXiv) Automated mathematical reasoning is a challenging problem that requires an agent to learn algebraic patterns that contain long-range dependencies. Two particular tasks that test this type of reasoning are (1) mathematical equation verification, which requires determining whether trigonometric and linear algebraic statements are valid identities or not, and (2) equation completion, which entails filling in a blank within an expression to make it true. Solving these tasks with deep learning requires that the neural model learn how to manipulate and compose various algebraic symbols, carrying this ability over to previously unseen expressions. Artificial neural networks, including recurrent networks and transformers, struggle to generalize on these kinds of difficult compositional problems, often exhibiting poor extrapolation performance. In contrast, recursive neural networks (recursive-NNs) are, theoretically, capable of achieving better extrapolation due to their tree-like design but are difficult to optimize as the depth of their underlying tree structure increases. To overcome this issue, we extend recursive-NNs to utilize multiplicative, higher-order synaptic connections and, furthermore, to learn to dynamically control and manipulate an external memory. We argue that this key modification gives the neural system the ability to capture powerful transition functions for each possible input. We demonstrate the effectiveness of our proposed higher-order, memory-augmented recursive-NN models on two challenging mathematical equation tasks, showing improved extrapolation, stable performance, and faster convergence. Our models achieve a 1.53% average improvement over current state-of-the-art methods in equation verification and achieve a 2.22% Top-1 average accuracy and 2.96% Top-5 average accuracy for equation completion.