# Never-Ending Behavior-Cloning Agent for Robotic Manipulation

arXiv: https://arxiv.org/abs/2403.00336 · HTML: https://mlsanity.com/paper/2403.00336/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 75 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA](https://mlsanity.com/paper/2304.06027/) (2023)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)

## Built on by (most cited first, 10 of 10)

- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model](https://mlsanity.com/paper/2511.01571/) (2025)
- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [Lifelong Language-Conditioned Robotic Manipulation Learning](https://mlsanity.com/paper/2603.05160/) (2026)
- [Lifelong Embodied Navigation Learning](https://mlsanity.com/paper/2603.06073/) (2026)
- [T2S: Tokenized Skill Scaling for Lifelong Imitation Learning](https://mlsanity.com/paper/2508.01167/) (2025)
- [All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation](https://mlsanity.com/paper/2603.14276/) (2026)
- [Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation](https://mlsanity.com/paper/2607.14852/) (2026)
- [Continual Hand-Eye Calibration for Open-world Robotic Manipulation](https://mlsanity.com/paper/2604.15814/) (2026)
- [FocalPolicy: Frequency-Optimized Chunking and Locally Anchored Flow Matching for Coherent Visuomotor Policy](https://mlsanity.com/paper/2605.15944/) (2026)

## Abstract (arXiv)

Relying on multi-modal observations, embodied robots (e.g., humanoid robots) could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents in robots still face existing long-standing challenges, i.e., 3D scene representation and human-level task learning, when adapting into a series of new tasks in practical scenarios. We here investigate these above challenges with NBAgent in embodied robots, a pioneering language-conditioned Never-ending Behavior-cloning Agent, which can continually learn observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-shared semantic rendering module and a skill-shared representation distillation module to effectively learn 3D scene semantics from skill-shared attribute, further tackling 3D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like human from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method.