# Training with Product Digital Twins for AutoRetail Checkout

arXiv: https://arxiv.org/abs/2308.09708 · HTML: https://mlsanity.com/paper/2308.09708/

## Key facts

- Cited by 4 papers in the map (data through 7 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Semantic Foggy Scene Understanding with Synthetic Data](https://mlsanity.com/paper/1708.07819/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in vitro](https://mlsanity.com/paper/1701.07717/) (2017)
- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Dissecting Person Re-identification from the Viewpoint of Viewpoint](https://mlsanity.com/paper/1812.02162/) (2018)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [Learning To Simulate](https://mlsanity.com/paper/1810.02513/) (2018)
- [Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining](https://mlsanity.com/paper/2107.14572/) (2021)
- [Simulating Content Consistent Vehicle Datasets with Attribute Descent](https://mlsanity.com/paper/1912.08855/) (2019)
- [RPC: A Large-Scale Retail Product Checkout Dataset](https://mlsanity.com/paper/1901.07249/) (2019)
- [M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining](https://mlsanity.com/paper/2109.04275/) (2021)
- [The 6th AI City Challenge](https://mlsanity.com/paper/2204.10380/) (2022)
- [Leveraging Weakly Annotated Data for Fashion Image Retrieval and Label Prediction](https://mlsanity.com/paper/1709.09426/) (2017)
- [Face-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/1909.01064/) (2019)

## Built on by (most cited first, 4 of 4)

- [ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2311.16494/) (2023)
- [More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.25848/) (2025)
- [Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic](https://mlsanity.com/paper/2310.04416/) (2023)
- [All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models](https://mlsanity.com/paper/2604.00479/) (2026)

## Abstract (arXiv)

Automating the checkout process is important in smart retail, where users effortlessly pass products by hand through a camera, triggering automatic product detection, tracking, and counting. In this emerging area, due to the lack of annotated training data, we introduce a dataset comprised of product 3D models, which allows for fast, flexible, and large-scale training data generation through graphic engine rendering. Within this context, we discern an intriguing facet, because of the user "hands-on" approach, bias in user behavior leads to distinct patterns in the real checkout process. The existence of such patterns would compromise training effectiveness if training data fail to reflect the same. To address this user bias problem, we propose a training data optimization framework, i.e., training with digital twins (DtTrain). Specifically, we leverage the product 3D models and optimize their rendering viewpoint and illumination to generate "digital twins" that visually resemble representative user images. These digital twins, inherit product labels and, when augmented, form the Digital Twin training set (DT set). Because the digital twins individually mimic user bias, the resulting DT training set better reflects the characteristics of the target scenario and allows us to train more effective product detection and tracking models. In our experiment, we show that DT set outperforms training sets created by existing dataset synthesis methods in terms of counting accuracy. Moreover, by combining DT set with pseudo-labeled real checkout data, further improvement is observed. The code is available at https://github.com/yorkeyao/Automated-Retail-Checkout.