mlsanityData to 14 Oct 2026

Modulating early visual processing by language

arXiv 1707.00683 ↗First posted 2 Jul 2017cs.CV cs.CL cs.LGcited by 167

Not on the map. Why?

Abstract (arXiv)

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the entire visual processing by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

Key facts. Cited by 167 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 60; 2022: 37; 2023: 25; 2024: 19; 2025: 9; 2026: 16.

167 papers.

Sort:

By area:

  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  2. 2021Dynamic Neural Networks: A Surveycited by 408
  3. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  4. 2021VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalizationcited by 287
  5. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  6. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  7. 2021LEAP: Learning Articulated Occupancy of Peoplecited by 107
  8. 2021Colorization Transformercited by 89
  9. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 83
  10. 2023Modular Deep Learningcited by 83
  11. 2022VIDM: Video Implicit Diffusion Modelscited by 78
  12. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
38 more
  1. 2021Region-aware Adaptive Instance Normalization for Image Harmonizationcited by 71
  2. 2021Learning a Universal Template for Few-shot Dataset Generalizationcited by 61
  3. 2021Rebooting ACGAN: Auxiliary Classifier GANs with Stable Trainingcited by 57
  4. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 53
  5. 2024CViT: Continuous Vision Transformer for Operator Learningcited by 51
  6. 2022Transfer without Forgettingcited by 44
  7. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 44
  8. 2022DrapeNet: Garment Generation and Self-Supervised Drapingcited by 41
  9. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 40
  10. 2022StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesiscited by 38
  11. 2023Diffusion-based Conditional ECG Generation with Structured State Space Modelscited by 36
  12. 2023PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervisioncited by 36
  13. 2021Selective Synthetic Augmentation with HistoGAN for Improved Histopathology Image Classificationcited by 35
  14. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 32
  15. 2021Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstructioncited by 31
  16. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replaycited by 31
  17. 2021PcDGAN: A Continuous Conditional Diverse Generative Adversarial Network For Inverse Designcited by 30
  18. 2023Lossless Adaptation of Pretrained Vision Models For Robotic Manipulationcited by 29
  19. 2022I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classificationcited by 27
  20. 2023Active Prompt Learning in Vision Language Modelscited by 25
  21. 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
  22. 2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
  23. 2022Out-of-Distribution Detection with Semantic Mismatch under Maskingcited by 22
  24. 2021The Power of Prompt Tuning for Low-Resource Semantic Parsingcited by 21
  25. 2021Conditional GANs with Auxiliary Discriminative Classifiercited by 20
  26. 2024GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Detailscited by 19
  27. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
  28. 2021Class Balancing GAN with a Classifier in the Loopcited by 17
  29. 2021DocSynth: A Layout Guided Approach for Controllable Document Image Synthesiscited by 17
  30. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  31. 2021Adaptive Denoising via GainTuningcited by 16
  32. 2021NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Droppingcited by 16
  33. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  34. 2021Semantic Palette: Guiding Scene Generation with Class Proportionscited by 14
  35. 2021Generative Modeling for Multi-task Visual Learningcited by 14
  36. 2022Improving GANs for Long-Tailed Data through Group Spectral Regularizationcited by 14
  37. 2021Efficient Conditional GAN Transfer with Knowledge Propagation across Classescited by 13
  38. 2022Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensingcited by 13

50 most cited of 167 All 167 (JSON)