Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models, offering flexible generation orders and strong performance on complex reasoning tasks. However, instruction-tuned dLLMs exhibit a critical vulnerability we term texttt<eos> overflow: as allocated sequence length increases, responses paradoxically become shorter, collapsing into early termination or degenerating into streams of texttt<eos> tokens. Although noticed in practice, this issue has not been systematically analyzed. We trace its root cause to the dual role of texttt<eos> as both termination and padding, which concentrates probability mass on texttt<eos> at later positions and propagates backward to trigger early termination. To address this, we introduce Rainbow Padding, a simple remedy that replaces repeated texttt<eos> placeholders with a repeating cycle of distinct padding tokens, distributing probability mass and breaking texttt<eos> dominance. Experiments show that Rainbow Padding substantially improves length robustness and output quality, with as few as seven padding tokens sufficient to prevent early termination. Moreover, the method integrates efficiently into existing instruction-tuned models: LoRA fine-tuning for a single epoch on minimal data yields significant improvements, making this solution highly practical. The code is publicly available at https://github.com/quasar529/rainbow-padding.