# Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement

arXiv: https://arxiv.org/abs/2212.05970 · HTML: https://mlsanity.com/paper/2212.05970/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 9 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models](https://mlsanity.com/paper/2102.04411/) (2021)
- [Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules](https://mlsanity.com/paper/2110.07720/) (2021)
- [Recursive Sketches for Modular Deep Learning](https://mlsanity.com/paper/1905.12730/) (2019)
- [AdapNet: Adaptability Decomposing Encoder-Decoder Network for Weakly Supervised Action Recognition and Localization](https://mlsanity.com/paper/1911.11961/) (2019)
- [Automating the Removal of Obsolete TODO Comments](https://mlsanity.com/paper/2108.05846/) (2021)
- [HSD-CNN: Hierarchically self decomposing CNN architecture using class specific filter sensitivity analysis](https://mlsanity.com/paper/1811.04406/) (2018)

## Built on by (most cited first, 3 of 3)

- [Scatter-Based Innovation Propagation in Large Language Models for Multi-Stage Process Adaptation](https://mlsanity.com/paper/2506.17949/) (2025)
- [Inferring Data Preconditions from Deep Learning Models for Trustworthy Prediction in Deployment](https://mlsanity.com/paper/2401.14628/) (2024)
- [NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models](https://mlsanity.com/paper/2508.11348/) (2025)

## Abstract (arXiv)

Can we take a recurrent neural network (RNN) trained to translate between languages and augment it to support a new natural language without retraining the model from scratch? Can we fix the faulty behavior of the RNN by replacing portions associated with the faulty behavior? Recent works on decomposing a fully connected neural network (FCNN) and convolutional neural network (CNN) into modules have shown the value of engineering deep models in this manner, which is standard in traditional SE but foreign for deep learning models. However, prior works focus on the image-based multiclass classification problems and cannot be applied to RNN due to (a) different layer structures, (b) loop structures, (c) different types of input-output architectures, and (d) usage of both nonlinear and logistic activation functions. In this work, we propose the first approach to decompose an RNN into modules. We study different types of RNNs, i.e., Vanilla, LSTM, and GRU. Further, we show how such RNN modules can be reused and replaced in various scenarios. We evaluate our approach against 5 canonical datasets (i.e., Math QA, Brown Corpus, Wiki-toxicity, Clinc OOS, and Tatoeba) and 4 model variants for each dataset. We found that decomposing a trained model has a small cost (Accuracy: -0.6%, BLEU score: +0.10%). Also, the decomposed modules can be reused and replaced without needing to retrain.