# Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning

arXiv: https://arxiv.org/abs/2208.06551 · HTML: https://mlsanity.com/paper/2208.06551/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 56 references that are in the map.
- First posted on arXiv 13 Aug 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Synthesizer: Rethinking Self-Attention in Transformer Models](https://mlsanity.com/paper/2005.00743/) (2020)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)

## Built on by (most cited first, 18 of 18)

- [Tuning computer vision models with task rewards](https://mlsanity.com/paper/2302.08242/) (2023)
- [A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2304.13649/) (2023)
- [A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision](https://mlsanity.com/paper/2303.17376/) (2023)
- [Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions](https://mlsanity.com/paper/2503.05186/) (2025)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization](https://mlsanity.com/paper/2402.11940/) (2024)
- [DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles](https://mlsanity.com/paper/2503.03651/) (2025)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Learning Human-Human Interactions in Images from Weak Textual Supervision](https://mlsanity.com/paper/2304.14104/) (2023)
- [GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning](https://mlsanity.com/paper/2505.21863/) (2025)
- [A request for clarity over the End of Sequence token in the Self-Critical Sequence Training](https://mlsanity.com/paper/2305.12254/) (2023)
- [Bidirectional Awareness Induction in Autoregressive Seq2Seq Models](https://mlsanity.com/paper/2408.13959/) (2024)
- [Shifted Window Fourier Transform And Retention For Image Captioning](https://mlsanity.com/paper/2408.13963/) (2024)
- [Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation](https://mlsanity.com/paper/2312.15872/) (2023)
- [A parameter-efficient three-branch architecture for multimodal misinformation detection with limited annotations](https://mlsanity.com/paper/2512.20257/) (2025)
- [Visual moral inference and communication](https://mlsanity.com/paper/2504.11473/) (2025)
- [SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning](https://mlsanity.com/paper/2507.12845/) (2025)
- [Diffusion Is Your Friend in Show, Suggest and Tell](https://mlsanity.com/paper/2512.10038/) (2025)

## Abstract (arXiv)

We introduce a method called the Expansion mechanism that processes the input unconstrained by the number of elements in the sequence. By doing so, the model can learn more effectively compared to traditional attention-based approaches. To support this claim, we design a novel architecture ExpansionNet v2 that achieved strong results on the MS COCO 2014 Image Captioning challenge and the State of the Art in its respective category, with a score of 143.7 CIDErD in the offline test split, 140.8 CIDErD in the online evaluation server and 72.9 AllCIDEr on the nocaps validation set. Additionally, we introduce an End to End training algorithm up to 2.8 times faster than established alternatives. Source code available at: https://github.com/jchenghu/ExpansionNet_v2