mlsanityData to 7 Oct 2026

X-Linear Attention Networks for Image Captioning

arXiv 2003.14080 ↗First posted 31 Mar 2020cs.CVcited by 198

Not on the map. Why?

Abstract (arXiv)

Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2nd order interactions across multi-modal inputs. Nevertheless, there has not been evidence in support of building such interactions concurrently with attention mechanism for image captioning. In this paper, we introduce a unified attention block -- X-Linear attention block, that fully employs bilinear pooling to selectively capitalize on visual information or perform multi-modal reasoning. Technically, X-Linear attention block simultaneously exploits both the spatial and channel-wise bilinear attention distributions to capture the 2nd order interactions between the input single-modal or multi-modal features. Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively. Furthermore, we present X-Linear Attention Networks (dubbed as X-LAN) that novelly integrates X-Linear attention block(s) into image encoder and sentence decoder of image captioning model to leverage higher order intra- and inter-modal interactions. The experiments on COCO benchmark demonstrate that our X-LAN obtains to-date the best published CIDEr performance of 132.0% on COCO Karpathy test split. When further endowing Transformer with X-Linear attention blocks, CIDEr is boosted up to 132.8%. Source code is available at urlhttps://github.com/Panda-Peter/image-captioning.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 71; 2023: 37; 2024: 22; 2025: 18; 2026: 2.

198 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  3. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  4. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  5. 2022Translation between Molecules and Natural Languagecited by 173
  6. 2022A Unified Sequence Interface for Vision Taskscited by 152
  7. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  8. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  11. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  12. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
38 more
  1. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  2. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  3. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  4. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  5. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  6. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  7. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  8. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  9. 2021The Dawn of Quantum Natural Language Processingcited by 60
  10. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  11. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 58
  12. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  13. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 53
  14. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  15. 2022NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Modelscited by 47
  16. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  17. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  18. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  19. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 42
  20. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  21. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
  22. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  23. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  24. 2022End-to-End Transformer Based Model for Image Captioningcited by 37
  25. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  26. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  27. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  28. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  29. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  30. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  31. 2022Dual Vision Transformercited by 29
  32. 2021TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learningcited by 28
  33. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  34. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  35. 2022CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognitioncited by 24
  36. 2022Prompt-based Learning for Unpaired Image Captioningcited by 23
  37. 2021CIDEr-R: Robust Consensus-based Image Description Evaluationcited by 22
  38. 2022Noise-aware Learning from Web-crawled Image-Text Data for Image Captioningcited by 22

50 most cited of 198 All 198 (JSON)