mlsanityData to 13 Oct 2026

Towards Language-guided Visual Recognition via Dynamic Convolutions

arXiv 2110.08797 ↗First posted 17 Oct 2021cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

In this paper, we are committed to establishing an unified and end-to-end multi-modal network via exploring the language-guided visual recognition. To approach this target, we first propose a novel multi-modal convolution module called Language-dependent Convolution (LaConv). Its convolution kernels are dynamically generated based on natural language information, which can help extract differentiated visual features for different multi-modal examples. Based on the LaConv module, we further build the first fully language-driven convolution network, termed as LaConvNet, which can unify the visual recognition and multi-modal reasoning in one forward structure. To validate LaConv and LaConvNet, we conduct extensive experiments on four benchmark datasets of two vision-and-language tasks, i.e., visual question answering (VQA) and referring expression comprehension (REC). The experimental results not only shows the performance gains of LaConv compared to the existing multi-modal modules, but also witness the merits of LaConvNet as an unified network, including compact network, high generalization ability and excellent performance, e.g., +4.7% on RefCOCO+.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). Builds on 71 of its 102 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

71 of 102 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  9. 2017Mask R-CNNcited by 7,266
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,836
  12. 2018YOLOv3: An Incremental Improvementcited by 2,655
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  3. 2015VQA: Visual Question Answeringcited by 2,495
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,306
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,450
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
  9. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 898
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 430
  12. 2015Neural Module Networkscited by 376
  13. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 371
  14. 2017A simple neural network module for relational reasoningcited by 345
  15. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  16. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  17. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  18. 2018Bilinear Attention Networkscited by 278
  19. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  20. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  21. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  22. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  23. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  24. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  25. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  26. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  27. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  28. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  29. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  30. 2017Modulating early visual processing by languagecited by 167
  31. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  32. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  33. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  34. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  35. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  36. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  37. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  38. 2019Learning by Abstraction: The Neural State Machinecited by 113

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 4; 2025: 1; 2026: 4.

17 papers.

Sort:

By area:

  1. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  2. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  3. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  4. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 14
  5. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 11
  6. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  7. 2023NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learningcited by 8
  8. 2026GREx: Generalized Referring Expression Segmentation, Comprehension, and Generationcited by 6
  9. 2022PixelFolder: An Efficient Progressive Pixel Synthesis Network for Image Generationcited by 5
  10. 2022A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehensioncited by 5
  11. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Modelscited by 5
  12. 2025AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selectioncited by 5
5 more
  1. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  2. 2024Mamba Capsule Routing Towards Part-Whole Relational Camouflaged Object Detectioncited by 3
  3. 2023Towards Omni-supervised Referring Expression Segmentationcited by 1
  4. 2026WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shiftcited by 1
  5. 2024Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Modelscited by 0