# Lane2Seq: Towards Unified Lane Detection via Sequence Generation

arXiv: https://arxiv.org/abs/2402.17172 · HTML: https://mlsanity.com/paper/2402.17172/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Spatial As Deep: Spatial CNN for Traffic Scene Understanding](https://mlsanity.com/paper/1712.06080/) (2017)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection](https://mlsanity.com/paper/2010.12035/) (2020)
- [Ultra Fast Structure-aware Deep Lane Detection](https://mlsanity.com/paper/2004.11757/) (2020)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [End-to-end Lane Shape Prediction with Transformers](https://mlsanity.com/paper/2011.04233/) (2020)
- [PolyLaneNet: Lane Estimation via Deep Polynomial Regression](https://mlsanity.com/paper/2004.10924/) (2020)
- [CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution](https://mlsanity.com/paper/2105.05003/) (2021)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)
- [Rethinking Efficient Lane Detection via Curve Modeling](https://mlsanity.com/paper/2203.02431/) (2022)
- [YOLOP: You Only Look Once for Panoptic Driving Perception](https://mlsanity.com/paper/2108.11250/) (2021)
- [CLRNet: Cross Layer Refinement Network for Lane Detection](https://mlsanity.com/paper/2203.10350/) (2022)
- [RESA: Recurrent Feature-Shift Aggregator for Lane Detection](https://mlsanity.com/paper/2008.13719/) (2020)
- [A Keypoint-based Global Association Network for Lane Detection](https://mlsanity.com/paper/2204.07335/) (2022)
- [SPTS: Single-Point Text Spotting](https://mlsanity.com/paper/2112.07917/) (2021)
- [Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.10334/) (2019)
- [LaneNet: Real-Time Lane Detection Networks for Autonomous Driving](https://mlsanity.com/paper/1807.01726/) (2018)
- [Tuning computer vision models with task rewards](https://mlsanity.com/paper/2302.08242/) (2023)

## Built on by (most cited first, 5 of 5)

- [Monocular Lane Detection Based on Deep Learning: A Survey](https://mlsanity.com/paper/2411.16316/) (2024)
- [From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving](https://mlsanity.com/paper/2509.23641/) (2025)
- [DiffusionLane: Diffusion Model for Lane Detection](https://mlsanity.com/paper/2510.22236/) (2025)
- [SIGMA-Lane: Scale-pyramId Gated MAmba for Temporally Consistent Video Lane Detection](https://mlsanity.com/paper/2608.16338/) (2026)
- [Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics](https://mlsanity.com/paper/2608.21440/) (2026)

## Abstract (arXiv)

In this paper, we present a novel sequence generation-based framework for lane detection, called Lane2Seq. It unifies various lane detection formats by casting lane detection as a sequence generation task. This is different from previous lane detection methods, which depend on well-designed task-specific head networks and corresponding loss functions. Lane2Seq only adopts a plain transformer-based encoder-decoder architecture with a simple cross-entropy loss. Additionally, we propose a new multi-format model tuning based on reinforcement learning to incorporate the task-specific knowledge into Lane2Seq. Experimental results demonstrate that such a simple sequence generation paradigm not only unifies lane detection but also achieves competitive performance on benchmarks. For example, Lane2Seq gets 97.95% and 97.42% F1 score on Tusimple and LLAMAS datasets, establishing a new state-of-the-art result for two benchmarks.