# TransFlow: Transformer as Flow Learner

arXiv: https://arxiv.org/abs/2304.11523 · HTML: https://mlsanity.com/paper/2304.11523/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 61 references that are in the map.
- First posted on arXiv 23 Apr 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [AANet: Adaptive Aggregation Network for Efficient Stereo Matching](https://mlsanity.com/paper/2004.09548/) (2020)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Learning Feature Descriptors using Camera Pose Supervision](https://mlsanity.com/paper/2004.13324/) (2020)
- [FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting](https://mlsanity.com/paper/2109.02974/) (2021)
- [Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://mlsanity.com/paper/1812.06264/) (2018)
- [Sequence Level Semantics Aggregation for Video Object Detection](https://mlsanity.com/paper/1907.06390/) (2019)

## Built on by (most cited first, 30 of 32)

- [SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow](https://mlsanity.com/paper/2405.14793/) (2024)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework](https://mlsanity.com/paper/2503.14880/) (2025)
- [StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences](https://mlsanity.com/paper/2311.17099/) (2023)
- [Rethinking RAFT for Efficient Optical Flow](https://mlsanity.com/paper/2401.00833/) (2024)
- [ClusterFormer: Clustering As A Universal Visual Learner](https://mlsanity.com/paper/2309.13196/) (2023)
- [Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera](https://mlsanity.com/paper/2307.06003/) (2023)
- [Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks](https://mlsanity.com/paper/2406.05857/) (2024)
- [Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow](https://mlsanity.com/paper/2401.17642/) (2024)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation](https://mlsanity.com/paper/2409.17001/) (2024)
- [BadPart: Unified Black-box Adversarial Patch Attacks against Pixel-wise Regression Tasks](https://mlsanity.com/paper/2404.00924/) (2024)
- [CMR-Agent: Learning a Cross-Modal Agent for Iterative Image-to-Point Cloud Registration](https://mlsanity.com/paper/2408.02394/) (2024)
- [Towards Dependable Optical Flow: Benchmarks for Real-World Spatiotemporal Corruptions](https://mlsanity.com/paper/2411.14865/) (2024)
- [FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures](https://mlsanity.com/paper/2310.20593/) (2023)
- [Independently Keypoint Learning for Small Object Semantic Correspondence](https://mlsanity.com/paper/2404.02678/) (2024)
- [A Survey on Video Analytics in Cloud-Edge-Terminal Collaborative Systems](https://mlsanity.com/paper/2502.06581/) (2025)
- [MambaFlow: A Mamba-Centric Architecture for End-to-End Optical Flow Estimation](https://mlsanity.com/paper/2503.07046/) (2025)
- [ProbDiffFlow: An Efficient Learning-Free Framework for Probabilistic Single-Image Optical Flow Estimation](https://mlsanity.com/paper/2503.12348/) (2025)
- [Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling](https://mlsanity.com/paper/2504.05537/) (2025)
- [Radiance Field Learners As UAV First-Person Viewers](https://mlsanity.com/paper/2408.05533/) (2024)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards](https://mlsanity.com/paper/2603.26378/) (2026)
- [Lightweight Improved Residual Network for Efficient Inverse Tone Mapping](https://mlsanity.com/paper/2307.03998/) (2023)
- [Occlusion-Aware 3D Motion Interpretation for Abnormal Behavior Detection](https://mlsanity.com/paper/2407.16788/) (2024)
- [Leveraging Object Priors for Point Tracking](https://mlsanity.com/paper/2409.05786/) (2024)
- [Content-Aware Preserving Image Generation](https://mlsanity.com/paper/2411.09871/) (2024)
- [Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining](https://mlsanity.com/paper/2504.20800/) (2025)
- [Intrinsic Dimensionality as a Model-Free Measure of Class Imbalance](https://mlsanity.com/paper/2511.10475/) (2025)
- [Flow and Depth Assisted Video Prediction with Latent Transformer](https://mlsanity.com/paper/2511.16484/) (2025)

## Abstract (arXiv)

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. In this work, we propose TransFlow, a pure transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g., occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it enables a concise self-learning paradigm and effectively eliminate the complex and laborious multi-stage pre-training procedures. We achieve the state-of-the-art results on the Sintel, KITTI-15, as well as several downstream tasks, including video object detection, interpolation and stabilization. For its efficacy, we hope TransFlow could serve as a flexible baseline for optical flow estimation.