Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation

Foundation models applied in robotics, particularly Vision--Language--Action (VLA) models, hold great promise for achieving general-purpose manipulation. Yet, systematic real-world evaluations and cross-model comparisons remain scarce. This paper reports our empirical experiences from benchmarking four representative VLAs -- ACT, OpenVLA--OFT, RDT-1B, and boldmathπ0 -- across four manipulation tasks conducted in both simulation and on the ALOHA Mobile platform. We establish a standardized evaluation framework that measures performance along three key dimensions: (1) textitaccuracy and efficiency (success rate and time-to-success), (2) textitadaptability across in-distribution, spatial out-of-distribution, and instance-plus-spatial out-of-distribution settings, and (3) textitlanguage instruction-following accuracy. Through this process, we observe that boldmathπ0 demonstrates superior adaptability in out-of-distribution scenarios, while ACT provides the highest stability in-distribution. Further analysis highlights differences in computational demands, data-scaling behavior, and recurring failure modes such as near-miss grasps, premature releases, and long-horizon state drift. These findings reveal practical trade-offs among VLA model architectures in balancing precision, generalization, and deployment cost, offering actionable insights for selecting and deploying VLAs in real-world robotic manipulation tasks.