# Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding

arXiv: https://arxiv.org/abs/2505.07600 · HTML: https://mlsanity.com/paper/2505.07600/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 31 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Low-Rank Few-Shot Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18541/) (2024)
- [MeshUDF: Fast and Differentiable Meshing of Unsigned Distance Field Networks](https://mlsanity.com/paper/2111.14549/) (2021)
- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World](https://mlsanity.com/paper/2312.02976/) (2023)
- [ElasticTok: Adaptive Tokenization for Image and Video](https://mlsanity.com/paper/2410.08368/) (2024)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling](https://mlsanity.com/paper/2410.18912/) (2024)
- [Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2502.16707/) (2025)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [GarmentTracking: Category-Level Garment Pose Tracking](https://mlsanity.com/paper/2303.13913/) (2023)
- [Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics](https://mlsanity.com/paper/2303.01310/) (2023)
- [Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data](https://mlsanity.com/paper/2401.01734/) (2024)
- [Leveraging Large Language Models for Multimodal Search](https://mlsanity.com/paper/2404.15790/) (2024)
- [Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention](https://mlsanity.com/paper/2301.03003/) (2023)
- [Reconstruction of Manipulated Garment with Guided Deformation Prior](https://mlsanity.com/paper/2405.10934/) (2024)
- [TRTM: Template-based Reconstruction and Target-oriented Manipulation of Crumpled Cloths](https://mlsanity.com/paper/2308.04670/) (2023)
- [BiFold: Bimanual Cloth Folding with Language Guidance](https://mlsanity.com/paper/2501.16458/) (2025)
- [Deformable Surface Reconstruction via Riemannian Metric Preservation](https://mlsanity.com/paper/2212.11596/) (2022)

## Abstract (arXiv)

Manipulating clothes is challenging due to their complex dynamics, high deformability, and frequent self-occlusions. Garments exhibit a nearly infinite number of configurations, making explicit state representations difficult to define. In this paper, we analyze BiFold, a model that predicts language-conditioned pick-and-place actions from visual observations, while implicitly encoding garment state through end-to-end learning. To address scenarios such as crumpled garments or recovery from failed manipulations, BiFold leverages temporal context to improve state estimation. We examine the internal representations of the model and present evidence that its fine-tuning and temporal context enable effective alignment between text and image regions, as well as temporal consistency.