# Controlling Neural Networks with Rule Representations

arXiv: https://arxiv.org/abs/2106.07804 · HTML: https://mlsanity.com/paper/2106.07804/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Boolean Decision Rules via Column Generation](https://mlsanity.com/paper/1805.09901/) (2018)
- [Lagrangian Duality for Constrained Deep Learning](https://mlsanity.com/paper/2001.09394/) (2020)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention](https://mlsanity.com/paper/2009.14799/) (2020)
- [Newton vs the machine: solving the chaotic three-body problem using deep neural networks](https://mlsanity.com/paper/1910.07291/) (2019)

## Built on by (most cited first, 16 of 16)

- [On the Constrained Time-Series Generation Problem](https://mlsanity.com/paper/2307.01717/) (2023)
- [User-Interactive Offline Reinforcement Learning](https://mlsanity.com/paper/2205.10629/) (2022)
- [Knowledge-augmented Deep Learning and Its Applications: A Survey](https://mlsanity.com/paper/2212.00017/) (2022)
- [Learning with Explanation Constraints](https://mlsanity.com/paper/2303.14496/) (2023)
- [Bayesian Neural Networks with Domain Knowledge Priors](https://mlsanity.com/paper/2402.13410/) (2024)
- [Deep Grey-Box Modeling With Adaptive Data-Driven Models Toward Trustworthy Estimation of Theory-Driven Models](https://mlsanity.com/paper/2210.13103/) (2022)
- [Automatic Rule Induction for Interpretable Semi-Supervised Learning](https://mlsanity.com/paper/2205.09067/) (2022)
- [Do Machine Learning Models Learn Statistical Rules Inferred from Data?](https://mlsanity.com/paper/2303.01433/) (2023)
- [Rule-adhering synthetic data -- the lingua franca of learning](https://mlsanity.com/paper/2209.06679/) (2022)
- [DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation](https://mlsanity.com/paper/2406.00611/) (2024)
- [ESC-Rules: Explainable, Semantically Constrained Rule Sets](https://mlsanity.com/paper/2208.12523/) (2022)
- [Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain](https://mlsanity.com/paper/2309.17171/) (2023)
- [Bridging the Intent Gap: Knowledge-Enhanced Visual Generation](https://mlsanity.com/paper/2405.12538/) (2024)
- [Integrating Optimization Theory with Deep Learning for Wireless Network Design](https://mlsanity.com/paper/2412.08761/) (2024)
- [Rule By Example: Harnessing Logical Rules for Explainable Hate Speech Detection](https://mlsanity.com/paper/2307.12935/) (2023)
- [Physics-informed Diffusion Generative Model for Time-Series Data Synthesis in Dynamic Systems](https://mlsanity.com/paper/2608.10941/) (2026)

## Abstract (arXiv)

We propose a novel training method that integrates rules into deep learning, in a way the strengths of the rules are controllable at inference. Deep Neural Networks with Controllable Rule Representations (DeepCTRL) incorporates a rule encoder into the model coupled with a rule-based objective, enabling a shared representation for decision making. DeepCTRL is agnostic to data type and model architecture. It can be applied to any kind of rule defined for inputs and outputs. The key aspect of DeepCTRL is that it does not require retraining to adapt the rule strength -- at inference, the user can adjust it based on the desired operation point on accuracy vs. rule verification ratio. In real-world domains where incorporating rules is critical -- such as Physics, Retail and Healthcare -- we show the effectiveness of DeepCTRL in teaching rules for deep learning. DeepCTRL improves the trust and reliability of the trained models by significantly increasing their rule verification ratio, while also providing accuracy gains at downstream tasks. Additionally, DeepCTRL enables novel use cases such as hypothesis testing of the rules on data samples, and unsupervised adaptation based on shared rules between datasets.