mlsanityData to 7 Oct 2026

Control-oriented Clustering of Visual Latent Representation

arXiv 2410.05063 ↗First posted 7 Oct 2024cs.LG cs.CV cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

We initiate a study of the geometry of the visual representation space -- the information channel from the vision encoder to the action decoder -- in an image-based control pipeline learned from behavior cloning. Inspired by the phenomenon of neural collapse (NC) in image classification (arXiv:2008.08186), we empirically demonstrate the prevalent emergence of a similar law of clustering in the visual representation space. Specifically, in discrete image-based control (e.g., Lunar Lander), the visual representations cluster according to the natural discrete action labels; in continuous image-based control (e.g., Planar Pushing and Block Stacking), the clustering emerges according to "control-oriented" classes that are based on (a) the relative pose between the object and the target in the input or (b) the relative pose of the object induced by expert actions in the output. Each of the classes corresponds to one relative pose orthant (REPO). Beyond empirical observation, we show such a law of clustering can be leveraged as an algorithmic tool to improve test-time performance when training a policy with limited expert demonstrations. Particularly, we pretrain the vision encoder using NC as a regularization to encourage control-oriented clustering of the visual features. Surprisingly, such an NC-pretrained vision encoder, when finetuned end-to-end with the action decoder, boosts the test-time performance by 10% to 35%. Real-world vision-based planar pushing experiments confirmed the surprising advantage of control-oriented visual representation pretraining.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 45 of its 71 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

45 of 71 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2016OpenAI Gymcited by 1,684
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
33 more
  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2018Behavioral Cloning from Observationcited by 482
  7. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  8. 2024The Platonic Representation Hypothesiscited by 362
  9. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  10. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  12. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  13. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  14. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  15. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  16. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  17. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  18. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  19. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  20. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  21. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  22. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  23. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  24. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  25. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  26. 2016More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushingcited by 30
  27. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  28. 2022Optimization-Based Control for Dynamic Legged Robotscited by 21
  29. 2023Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networkscited by 20
  30. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  31. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
  32. 2016Feedback Control of the Pusher-Slider System: A Story of Hybrid and Underactuated Contact Dynamicscited by 13
  33. 2024Can We Understand Plasticity Through Neural Collapse?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  2. 2025MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoningcited by 9
  3. 2025Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splattingcited by 5
  4. 2025Compose by Focus: Scene Graph-based Atomic Skillscited by 4
  5. 2026DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representationcited by 4
  6. 2026Capturing Visual Environment Structure Correlates with Control Performancecited by 2
  7. 2026Universal Guideline-Driven Image Clustering via a Hybrid LLM Agentcited by 0
  8. 2026Ordinal Neural Collapse as a Representation Prior for Visual Navigationcited by 0