mlsanityData to 7 Oct 2026

Language-Driven Representation Learning for Robotics

arXiv 2302.12766 ↗First posted 24 Feb 2023cs.RO cs.AI cs.CLcited by 163

Not on the map. Why?

Cited by 29 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work in visual representation learning for robotics demonstrates the viability of learning from large video datasets of humans performing everyday tasks. Leveraging methods such as masked autoencoding and contrastive learning, these representations exhibit strong transfer to policy learning for visuomotor control. But, robot learning encompasses a diverse set of problems beyond control including grasp affordance prediction, language-conditioned imitation learning, and intent scoring for human-robot collaboration, amongst others. First, we demonstrate that existing representations yield inconsistent results across these tasks: masked autoencoding approaches pick up on low-level spatial features at the cost of high-level semantics, while contrastive learning approaches capture the opposite. We then introduce Voltron, a framework for language-driven representation learning from human videos and associated captions. Voltron trades off language-conditioned visual reconstruction to learn low-level visual patterns, and visually-grounded language generation to encode high-level semantics. We also construct a new evaluation suite spanning five distinct robot learning problems unicodex2013 a unified platform for holistically evaluating visual representations for robotics. Through comprehensive, controlled experiments across all five problems, we find that Voltron's language-driven representations outperform the prior state-of-the-art, especially on targeted problems requiring higher-level features.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Builds on 86 of its 113 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

86 of 113 references.

References · 86

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2020GLU Variants Improve Transformercited by 1,724
  13. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  18. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  19. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  21. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  22. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  23. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  24. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  25. 2016Modeling Context in Referring Expressionscited by 1,054
  26. 2021Scaling Vision Transformerscited by 993
  27. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  28. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  29. 2022A Generalist Agentcited by 802
  30. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  31. 2021Going deeper with Image Transformerscited by 759
  32. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  33. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  34. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  35. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  36. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  37. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  38. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 35; 2024: 61; 2025: 48; 2026: 19.

163 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  4. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  6. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  8. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  9. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  10. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  11. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  12. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
38 more
  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  3. 2024RT-H: Action Hierarchies Using Languagecited by 204
  4. 2023Large Language Models as General Pattern Machinescited by 196
  5. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  7. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  8. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  9. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  10. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  11. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  12. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  13. 2025GR-3 Technical Reportcited by 110
  14. 2023On Bringing Robots Homecited by 101
  15. 2024Vision Language Models are In-Context Value Learnerscited by 84
  16. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  17. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  18. 2023Data Quality in Imitation Learningcited by 72
  19. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  20. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  21. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  22. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  23. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  24. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  25. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  26. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  27. 2024Diffusion Transformer Policycited by 41
  28. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  29. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  30. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 38
  31. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  32. 2024Learning Manipulation by Predicting Interactioncited by 36
  33. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  34. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  35. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  36. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  37. 2023Interactive Natural Language Processingcited by 31
  38. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31

50 most cited of 163 All 163 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
  2. 2026V-Link: Recovering Lost Visual Representations in Action DiT for Vision-Language-Action Models
  3. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
  4. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  5. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Models
  6. 2025Contrastive Representation Regularization for Vision-Language-Action Models
  7. 2026LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
  8. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data