Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

Discrete diffusion language models (dLLMs) recover masked tokens in parallel, offering significant speedups over autoregressive (AR) generation. However, such promising frameworks face a fundamental architectural design dilemma: ding182 Adopting bidirectional attention achieves strong generation quality by allowing each position to access the full context, but is inherently incompatible with KV caching, limiting inference throughput in batch-serving scenarios; ding183 Conversely, causal attention enables efficient cached inference but loses all right-side context, substantially degrading generation quality. This paper introduces Bifocal dLLMs, a new paradigm that resolves this dilemma through asymmetric bidirectional context. Analogous to bifocal lenses, we instantiate the paradigm as R2LM (Right-to-Left Mamba), which combines two complementary mechanisms: a) standard causal attention providing precise left-context with full KV cache compatibility, while b) a lightweight reverse Mamba SSM sidecar supplying compressed right-side context without breaking cacheability. Comprehensive experiments on continued pretraining of Qwen3-1.7B with 60B tokens demonstrate that R2LM achieves 2.4× to 12.9× higher throughput than bidirectional dLLMs and 1.9× to 2.9× speedup over AR baselines in batch serving through parallel decoding with KV caching, while exceeding the causal baseline on most benchmarks and surpassing the bidirectional dLLM on average.