# Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

arXiv: https://arxiv.org/abs/2607.13522 · HTML: https://mlsanity.com/paper/2607.13522/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 60 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 30 of 55)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers](https://mlsanity.com/paper/2409.20537/) (2024)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [FLARE: Robot Learning with Implicit World Modeling](https://mlsanity.com/paper/2505.15659/) (2025)
- [PixNerd: Pixel Neural Field Diffusion](https://mlsanity.com/paper/2507.23268/) (2025)
- [VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers](https://mlsanity.com/paper/2507.01016/) (2025)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)

## Built on by (most cited first, 1 of 1)

- [What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations](https://mlsanity.com/paper/2607.27017/) (2026)

## Abstract (arXiv)

A robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and raw vision features read force at R2 at or below 0.10 on every robot we test. We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent with a learned-query cross-attention layer, trained self-supervised by masked cross-modal prediction under the LeJEPA/SIGReg objective. At evaluation only vision enters, which poses a sharp question. Does fusing state into training make the vision-only latent carry anything the pixels do not already contain? On the RH20T corpus the answer is yes, precisely where the camera is weakest. On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot, though absolute force recovery at a single timestep is modest; on motor state, which the camera largely sees, it is statistically tied with the strongest vision baselines, and it is the only feature whose latent geometry tracks state. A single embodiment-agnostic encoder covers four robots, and a data-matched control shows this breadth reflects embodiment diversity rather than data volume. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free invalid-state monitor (AUROC 0.90 on out-of-range states, 0.69 on scene-swapped states), and a diffusion decoder (PixNerd) reconstructs the camera frame from the latent, confirming the spatial compression preserves world-state. This report validates the single-timestep case; native-rate temporal fusion is the next step.