# Technical Report: Auxiliary Tuning and its Application to Conditional Text Generation

arXiv: https://arxiv.org/abs/2006.16823 · HTML: https://mlsanity.com/paper/2006.16823/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2407.12824/) (2024)
- [APPDIA: A Discourse-aware Transformer-based Style Transfer Model for Offensive Social Media Conversations](https://mlsanity.com/paper/2209.08207/) (2022)
- [ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models](https://mlsanity.com/paper/2503.22048/) (2025)
- [Automating Steering for Safe Multimodal Large Language Models](https://mlsanity.com/paper/2507.13255/) (2025)
- [Self-conditioning pre-trained language models](https://mlsanity.com/paper/2110.02802/) (2021)
- [Re-Imagining Multimodal Instruction Tuning: A Representation View](https://mlsanity.com/paper/2503.00723/) (2025)
- [Decoupling Task-Solving and Output Formatting in LLM Generation](https://mlsanity.com/paper/2510.03595/) (2025)
- [Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs](https://mlsanity.com/paper/2406.04460/) (2024)
- [Few-Shot Table-to-Text Generation with Prefix-Controlled Generator](https://mlsanity.com/paper/2208.10709/) (2022)
- [Function-Guided Conditional Generation Using Protein Language Models with Adapters](https://mlsanity.com/paper/2410.03634/) (2024)
- [Computational Storytelling and Emotions: A Survey](https://mlsanity.com/paper/2205.10967/) (2022)
- [Selective Token Generation for Few-shot Natural Language Generation](https://mlsanity.com/paper/2209.08206/) (2022)
- [Towards Lightweight, Adaptive and Attribute-Aware Multi-Aspect Controllable Text Generation with Large Language Models](https://mlsanity.com/paper/2502.13474/) (2025)
- [PAGE: Prompt Augmentation for text Generation Enhancement](https://mlsanity.com/paper/2510.13880/) (2025)
- [Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models](https://mlsanity.com/paper/2608.14797/) (2026)

## Abstract (arXiv)

We introduce a simple and efficient method, called Auxiliary Tuning, for adapting a pre-trained Language Model to a novel task; we demonstrate this approach on the task of conditional text generation. Our approach supplements the original pre-trained model with an auxiliary model that shifts the output distribution according to the target task. The auxiliary model is trained by adding its logits to the pre-trained model logits and maximizing the likelihood of the target task output. Our method imposes no constraints on the auxiliary architecture. In particular, the auxiliary model can ingest additional input relevant to the target task, independently from the pre-trained model's input. Furthermore, mixing the models at the logits level provides a natural probabilistic interpretation of the method. Our method achieved similar results to training from scratch for several different tasks, while using significantly fewer resources for training; we share a specific example of text generation conditioned on keywords.