# Deep Extrapolation for Attribute-Enhanced Generation

arXiv: https://arxiv.org/abs/2107.02968 · HTML: https://mlsanity.com/paper/2107.02968/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Delving into Deep Imbalanced Regression](https://mlsanity.com/paper/2102.09554/) (2021)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Model Inversion Networks for Model-Based Optimization](https://mlsanity.com/paper/1912.13464/) (2019)
- [Controlling Output Length in Neural Encoder-Decoders](https://mlsanity.com/paper/1609.09552/) (2016)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [Autofocused oracles for model-based design](https://mlsanity.com/paper/2006.08052/) (2020)
- [Optimizing Molecules using Efficient Queries from Property Evaluations](https://mlsanity.com/paper/2011.01921/) (2020)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)
- [Protein sequence design with deep generative models](https://mlsanity.com/paper/2104.04457/) (2021)
- [Design by adaptive sampling](https://mlsanity.com/paper/1810.03714/) (2018)
- [Adaptive machine learning for protein engineering](https://mlsanity.com/paper/2106.05466/) (2021)

## Built on by (most cited first, 18 of 18)

- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences](https://mlsanity.com/paper/2306.03111/) (2023)
- [Deep Generative Models for Detector Signature Simulation: A Taxonomic Review](https://mlsanity.com/paper/2312.09597/) (2023)
- [Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC](https://mlsanity.com/paper/2212.09925/) (2022)
- [Automating Steering for Safe Multimodal Large Language Models](https://mlsanity.com/paper/2507.13255/) (2025)
- [Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks](https://mlsanity.com/paper/2410.22296/) (2024)
- [Robust Guided Diffusion for Offline Black-Box Optimization](https://mlsanity.com/paper/2410.00983/) (2024)
- [Extrapolative Controlled Sequence Generation via Iterative Refinement](https://mlsanity.com/paper/2303.04562/) (2023)
- [Contrastive losses as generalized models of global epistasis](https://mlsanity.com/paper/2305.03136/) (2023)
- [Robust Model-Based Optimization for Challenging Fitness Landscapes](https://mlsanity.com/paper/2305.13650/) (2023)
- [TemperatureGAN: Generative Modeling of Regional Atmospheric Temperatures](https://mlsanity.com/paper/2306.17248/) (2023)
- [Graph Community Augmentation with GMM-based Modeling in Latent Space](https://mlsanity.com/paper/2412.01163/) (2024)
- [AffinityFlow: Guided Flows for Antibody Affinity Maturation](https://mlsanity.com/paper/2502.10365/) (2025)
- [Is Self-Supervised Pretraining Good for Extrapolation in Molecular Property Prediction?](https://mlsanity.com/paper/2308.08129/) (2023)
- [Towards Lightweight, Adaptive and Attribute-Aware Multi-Aspect Controllable Text Generation with Large Language Models](https://mlsanity.com/paper/2502.13474/) (2025)
- [Learning Extrapolative Sequence Transformations from Markov Chains](https://mlsanity.com/paper/2505.20251/) (2025)
- [An Information Theoretic Framework for Graph Novelty Generation via Latent Mixture Modeling](https://mlsanity.com/paper/2606.19770/) (2026)
- [Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models](https://mlsanity.com/paper/2608.14797/) (2026)

## Abstract (arXiv)

Attribute extrapolation in sample generation is challenging for deep neural networks operating beyond the training distribution. We formulate a new task for extrapolation in sequence generation, focusing on natural language and proteins, and propose GENhance, a generative framework that enhances attributes through a learned latent space. Trained on movie reviews and a computed protein stability dataset, GENhance can generate strongly-positive text reviews and highly stable protein sequences without being exposed to similar data during training. We release our benchmark tasks and models to contribute to the study of generative modeling extrapolation and data-driven design in biology and chemistry.