mlsanityData to 14 Oct 2026

ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning

arXiv 2306.00103 ↗First posted 31 May 2023cs.CV cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream VL tasks. Although the most advanced work improves performance by building bridges between encoders, it suffers from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of uni-modal semantic knowledge. In this work, we propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels. The managers introduced in each cross-modal layer can adaptively aggregate uni-modal semantic knowledge to facilitate more comprehensive cross-modal alignment and fusion. ManagerTower outperforms previous strong baselines both with and without Vision-Language Pre-training (VLP). With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. Code and checkpoints are available at https://github.com/LooperXX/ManagerTower.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

47 of 69 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2016Layer Normalizationcited by 5,669
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
35 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2018Deep contextualized word representationscited by 2,290
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  14. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  15. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  18. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  19. 2021Intriguing Properties of Vision Transformerscited by 446
  20. 2017Deep Layer Aggregationcited by 436
  21. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  22. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  23. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  24. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  25. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  26. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  27. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  28. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  29. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  30. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  31. 2021Go Wider Instead of Deepercited by 57
  32. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  33. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learningcited by 19
  34. 2020Multi-layer Representation Fusion for Neural Machine Translationcited by 15
  35. 2020Multiscale Collaborative Deep Models for Neural Machine Translationcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Copy-Move Forgery Detection and Question Answering for Remote Sensing Imagecited by 0
  2. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set
  2. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  3. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  4. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  5. 2025Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
  6. 2024Vision-Language Models Create Cross-Modal Task Representations
  7. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
  8. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks