mlsanityData to 7 Oct 2026

ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning

arXiv 2202.05451 ↗First posted 11 Feb 2022cs.CV cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent research that applies Transformer-based architectures to image captioning has resulted in state-of-the-art image captioning performance, capitalising on the success of Transformers on natural language tasks. Unfortunately, though these models work well, one major flaw is their large model sizes. To this end, we present three parameter reduction methods for image captioning Transformers: Radix Encoding, cross-layer parameter sharing, and attention parameter sharing. By combining these methods, our proposed ACORT models have 3.7x to 21.6x fewer parameters than the baseline model without compromising test performance. Results on the MS-COCO dataset demonstrate that our ACORT models are competitive against baselines and SOTA approaches, with CIDEr score >=126. Finally, we present qualitative results and ablation studies to demonstrate the efficacy of the proposed changes further. Code and pre-trained models are publicly available at https://github.com/jiahuei/sparse-image-captioning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 53 of its 62 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

53 of 62 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Layer Normalizationcited by 5,579
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
38 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  5. 2018Universal Transformerscited by 613
  6. 2019Deep Equilibrium Modelscited by 603
  7. 2016Self-critical Sequence Training for Image Captioningcited by 544
  8. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  9. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  10. 2019Attention on Attention for Image Captioningcited by 289
  11. 2016Using the Output Embedding to Improve Language Modelscited by 266
  12. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  13. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  14. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  15. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  16. 2020X-Linear Attention Networks for Image Captioningcited by 198
  17. 2019Levenshtein Transformercited by 197
  18. 2016Exploring the Limits of Language Modelingcited by 172
  19. 2019Image Captioning: Transforming Objects into Wordscited by 171
  20. 2014From Captions to Visual Concepts and Backcited by 109
  21. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  22. 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
  23. 2020Channel Pruning via Automatic Structure Searchcited by 84
  24. 2019Network Pruning via Transformable Architecture Searchcited by 80
  25. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  26. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  27. 2016Image Captioning and Visual Question Answering Based on Attributes and External Knowledgecited by 44
  28. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  29. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  30. 2019Fully Quantized Transformer for Machine Translationcited by 36
  31. 2019Sharing Attention Weights for Fast Transformercited by 34
  32. 2018Parameter Sharing Methods for Multilingual Self-Attentional Translation Modelscited by 33
  33. 2020A Better Variant of Self-Critical Sequence Trainingcited by 23
  34. 2018Grow and Prune Compact, Fast, and Accurate LSTMscited by 20
  35. 2017Learning Intrinsic Sparse Structures within Long Short-Term Memorycited by 18
  36. 2020DeLighT: Deep and Light-weight Transformercited by 17
  37. 2015Strategies for Training Large Vocabulary Neural Language Modelscited by 14
  38. 2019COMIC: Towards A Compact Image Captioning Model with Attentioncited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 2.

6 papers.

  1. 2023Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Imagescited by 2
  2. 2024Shifted Window Fourier Transform And Retention For Image Captioningcited by 2
  3. 2023TS-RGBD Dataset: a Novel Dataset for Theatre Scenes Description for People with Visual Impairmentscited by 1
  4. 2025Attention-based transformer models for image captioning across languages: An in-depth survey and evaluationcited by 1
  5. 2022Bypass Network for Semantics Driven Image Paragraph Captioningcited by 0
  6. 2025AC-Lite : A Lightweight Image Captioning Model for Low-Resource Assamese Languagecited by 0