# Encoder-Agnostic Adaptation for Conditional Language Generation

arXiv: https://arxiv.org/abs/1908.06938 · HTML: https://mlsanity.com/paper/1908.06938/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 27 of 27)

- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Transformer-based Conditional Variational Autoencoder for Controllable Story Generation](https://mlsanity.com/paper/2101.00828/) (2021)
- [LLM Augmented LLMs: Expanding Capabilities through Composition](https://mlsanity.com/paper/2401.02412/) (2024)
- [Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment](https://mlsanity.com/paper/2302.00902/) (2023)
- [Attribute Alignment: Controlling Text Generation from Pre-trained Language Models](https://mlsanity.com/paper/2103.11070/) (2021)
- [Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2205.15608/) (2022)
- [Medical Image Captioning via Generative Pretrained Transformers](https://mlsanity.com/paper/2209.13983/) (2022)
- [Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior](https://mlsanity.com/paper/2108.04812/) (2021)
- [Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models](https://mlsanity.com/paper/2211.08682/) (2022)
- [RepsNet: Combining Vision with Language for Automated Medical Reports](https://mlsanity.com/paper/2209.13171/) (2022)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes](https://mlsanity.com/paper/2408.11965/) (2024)
- [Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events](https://mlsanity.com/paper/2101.00822/) (2021)
- [Multimodal Conditionality for Natural Language Generation](https://mlsanity.com/paper/2109.01229/) (2021)
- [Clinical-grade Multi-Organ Pathology Report Generation for Multi-scale Whole Slide Images via a Semantically Guided Medical Text Foundation Model](https://mlsanity.com/paper/2409.15574/) (2024)
- [A Temporal Variational Model for Story Generation](https://mlsanity.com/paper/2109.06807/) (2021)
- [Hidden Schema Networks](https://mlsanity.com/paper/2207.03777/) (2022)
- [DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis](https://mlsanity.com/paper/2504.11082/) (2025)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation](https://mlsanity.com/paper/2405.14113/) (2024)
- [APT: Affine Prototype-Timestamp For Time Series Forecasting Under Distribution Shift](https://mlsanity.com/paper/2511.12945/) (2025)
- [XFBoost: Improving Text Generation with Controllable Decoders](https://mlsanity.com/paper/2202.08124/) (2022)
- [A Vision for Multisensory Intelligence: Sensing, Science, and Synergy](https://mlsanity.com/paper/2601.04563/) (2026)

## Abstract (arXiv)

Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However it is an open-question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. We therefore propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data efficient.