# Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

arXiv: https://arxiv.org/abs/2107.13077 · HTML: https://mlsanity.com/paper/2107.13077/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Controlling Output Length in Neural Encoder-Decoders](https://mlsanity.com/paper/1609.09552/) (2016)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)
- [Language Model Prior for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2004.14928/) (2020)
- [TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising](https://mlsanity.com/paper/2001.00725/) (2020)
- [Neural Response Generation with Meta-Words](https://mlsanity.com/paper/1906.06050/) (2019)
- [Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization](https://mlsanity.com/paper/1811.01100/) (2018)

## Built on by (most cited first, 4 of 4)

- [Automatic Rule Induction for Interpretable Semi-Supervised Learning](https://mlsanity.com/paper/2205.09067/) (2022)
- [DECIDER: A Dual-System Rule-Controllable Decoding Framework for Language Generation](https://mlsanity.com/paper/2403.01954/) (2024)
- [Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models](https://mlsanity.com/paper/2310.16343/) (2023)
- [ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation](https://mlsanity.com/paper/2608.22559/) (2026)

## Abstract (arXiv)

Sequence-to-Sequence (S2S) neural text generation models, especially the pre-trained ones (e.g., BART and T5), have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules (e.g., controllable constraints, prior knowledge) need to be executed. Previous works either design specific model structure (e.g., Copy Mechanism corresponding to the rule "the generated output should include certain words in the source input") or implement specialized inference algorithm (e.g., Constrained Beam Search) to execute particular rules through the text generation. These methods require careful design case-by-case and are difficult to support multiple rules concurrently. In this paper, we propose a novel module named Neural Rule-Execution Tracking Machine that can be equipped into various transformer-based generators to leverage multiple rules simultaneously to guide the neural generation model for superior generation performance in a unified and scalable way. Extensive experimental results on several benchmarks verify the effectiveness of our proposed model in both controllable and general text generation.