# ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning

arXiv: https://arxiv.org/abs/2202.05451 · HTML: https://mlsanity.com/paper/2202.05451/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 62 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)

## Built on by (most cited first, 6 of 6)

- [Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Images](https://mlsanity.com/paper/2308.13892/) (2023)
- [Shifted Window Fourier Transform And Retention For Image Captioning](https://mlsanity.com/paper/2408.13963/) (2024)
- [TS-RGBD Dataset: a Novel Dataset for Theatre Scenes Description for People with Visual Impairments](https://mlsanity.com/paper/2308.01035/) (2023)
- [Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation](https://mlsanity.com/paper/2506.05399/) (2025)
- [Bypass Network for Semantics Driven Image Paragraph Captioning](https://mlsanity.com/paper/2206.10059/) (2022)
- [AC-Lite : A Lightweight Image Captioning Model for Low-Resource Assamese Language](https://mlsanity.com/paper/2503.01453/) (2025)

## Abstract (arXiv)

Recent research that applies Transformer-based architectures to image captioning has resulted in state-of-the-art image captioning performance, capitalising on the success of Transformers on natural language tasks. Unfortunately, though these models work well, one major flaw is their large model sizes. To this end, we present three parameter reduction methods for image captioning Transformers: Radix Encoding, cross-layer parameter sharing, and attention parameter sharing. By combining these methods, our proposed ACORT models have 3.7x to 21.6x fewer parameters than the baseline model without compromising test performance. Results on the MS-COCO dataset demonstrate that our ACORT models are competitive against baselines and SOTA approaches, with CIDEr score >=126. Finally, we present qualitative results and ablation studies to demonstrate the efficacy of the proposed changes further. Code and pre-trained models are publicly available at https://github.com/jiahuei/sparse-image-captioning.