mlsanityData to 7 Oct 2026

Non-autoregressive Sequence-to-Sequence Vision-Language Models

arXiv 2403.02249 ↗First posted 4 Mar 2024cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Sequence-to-sequence vision-language models are showing promise, but their applicability is limited by their inference latency due to their autoregressive way of generating predictions. We propose a parallel decoding sequence-to-sequence vision-language model, trained with a Query-CTC loss, that marginalizes over multiple inference paths in the decoder. This allows us to model the joint distribution of tokens, rather than restricting to conditional distribution as in an autoregressive model. The resulting model, NARVL, achieves performance on-par with its state-of-the-art autoregressive counterpart, but is faster at inference time, reducing from the linear complexity associated with the sequential generation of tokens to a paradigm of constant time joint inference.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 59 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
34 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2017Non-Autoregressive Neural Machine Translationcited by 457
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  9. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  13. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  14. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  15. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  16. 2019Levenshtein Transformercited by 197
  17. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  18. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  19. 2022A Unified Sequence Interface for Vision Taskscited by 152
  20. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  21. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  22. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  23. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  24. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  25. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  26. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  27. 2019Fast Structured Decoding for Sequence Modelscited by 59
  28. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  29. 2018Semi-Autoregressive Neural Machine Translationcited by 48
  30. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  31. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
  32. 2019Learning Instance Occlusion for Panoptic Segmentationcited by 30
  33. 2020Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translationcited by 3
  34. 2023Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Promptscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  2. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  3. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  4. 2024COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinementcited by 3
  5. 2026Efficient Document Parsing via Parallel Token Predictioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation
  2. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  3. 2025FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
  4. 2024Masked AutoDecoder is Effective Multi-Task Vision Generalist
  5. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
  6. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
  7. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  8. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models