# Latent Flow Transformer

arXiv: https://arxiv.org/abs/2505.14513 · HTML: https://mlsanity.com/paper/2505.14513/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [One Step Diffusion via Shortcut Models](https://mlsanity.com/paper/2410.12557/) (2024)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [On the Effect of Dropping Layers of Pre-trained Transformer Models](https://mlsanity.com/paper/2004.03844/) (2020)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise](https://mlsanity.com/paper/2212.11685/) (2022)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling](https://mlsanity.com/paper/2402.10635/) (2024)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Reassessing Layer Pruning in LLMs: New Insights and Methods](https://mlsanity.com/paper/2411.15558/) (2024)
- [Simulation-Free Training of Neural ODEs on Paired Data](https://mlsanity.com/paper/2410.22918/) (2024)
- [Flowing Through Layers: A Continuous Dynamical Systems Perspective on Transformers](https://mlsanity.com/paper/2502.05656/) (2025)

## Built on by (most cited first, 1 of 1)

- [Difficulty-Calibrated Interpolation Paths for Conditional Flow Matching](https://mlsanity.com/paper/2608.21286/) (2026)

## Abstract (arXiv)

Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While more layers can lead to better performance, this approach has been challenged as far from efficient, especially given the superiority of continuous layers demonstrated by diffusion and flow-based models for image generation. We propose the Latent Flow Transformer (LFT), which replaces a block of layers with a single learned transport operator trained via flow matching, offering significant compression while maintaining compatibility with the original architecture. Additionally, we address the limitations of existing flow-based methods in textitpreserving coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M model, LFT trained with flow matching compresses 6 of 24 layers and outperforms directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529), demonstrating the feasibility of this design. When trained with FW, LFT further distills 12 layers into one while reducing the KL to 0.736 surpassing that from skipping 3 layers (0.932), significantly narrowing the gap between autoregressive and flow-based generation paradigms.