mlsanityData to 7 Oct 2026

Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Control

arXiv 2406.06072 ↗First posted 10 Jun 2024cs.CV cs.LG cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT), when paired with large-scale pretraining, have shown remarkable performance across various computer vision tasks, primarily due to their weak inductive bias. However, while such weak inductive bias aids in pretraining scalability, this may hinder the effective adaptation of ViTs for visuo-motor control tasks as a result of the absence of control-centric inductive biases. Such absent inductive biases include spatial locality and translation equivariance bias which convolutions naturally offer. To this end, we introduce Convolution Injector (CoIn), an add-on module that injects convolutions which are rich in locality and equivariance biases into a pretrained ViT for effective adaptation in visuo-motor control. We evaluate CoIn with three distinct types of pretrained ViTs (CLIP, MVP, VC-1) across 12 varied control tasks within three separate domains (Adroit, MetaWorld, DMC), and demonstrate that CoIn consistently enhances control task performance across all experimented environments and models, validating the effectiveness of providing pretrained ViTs with control-centric biases.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 65 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

59 of 65 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Segment Anythingcited by 8,158
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Mask R-CNNcited by 7,259
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2017Deformable Convolutional Networkscited by 1,481
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2018DeepMind Control Suitecited by 935
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  9. 2018Panoptic Segmentationcited by 796
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  14. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  15. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  16. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  17. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  18. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  19. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  20. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  21. 2021Intriguing Properties of Vision Transformerscited by 446
  22. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  23. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  24. 2022How Do Vision Transformers Work?cited by 305
  25. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  26. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  27. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  28. 2021Conformer: Local Features Coupling Global Representations for Visual Recognitioncited by 164
  29. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  30. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  31. 2021Towards Robust Vision Transformercited by 137
  32. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  33. 2022Inception Transformercited by 99
  34. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  35. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  36. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  37. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  38. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.