# OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

arXiv: https://arxiv.org/abs/2603.02390 · HTML: https://mlsanity.com/paper/2603.02390/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023)
- [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022)
- [Deep ConvLSTM with self-attention for human activity decoding using wearables](https://mlsanity.com/paper/2005.00698/) (2020)
- [HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding](https://mlsanity.com/paper/2307.05721/) (2023)
- [CenterFace: Joint Face Detection and Alignment Using Face as Point](https://mlsanity.com/paper/1911.03599/) (2019)
- [OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments](https://mlsanity.com/paper/2212.11152/) (2022)
- [Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs](https://mlsanity.com/paper/2406.01316/) (2024)
- [InMyFace: Inertial and Mechanomyography-Based Sensor Fusion for Wearable Facial Activity Recognition](https://mlsanity.com/paper/2302.04024/) (2023)
- [TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines](https://mlsanity.com/paper/2408.14146/) (2024)
- [Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion](https://mlsanity.com/paper/2501.07408/) (2025)
- [Unimodal and Multimodal Sensor Fusion for Wearable Activity Recognition](https://mlsanity.com/paper/2404.16005/) (2024)

## Built on by (most cited first, 2 of 2)

- [ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding](https://mlsanity.com/paper/2604.00767/) (2026)
- [HEIR: Learning Human-Entity Interactions with Functional Roles](https://mlsanity.com/paper/2609.35955/) (2026)

## Abstract (arXiv)

Smart factories use advanced technologies to optimize production and increase efficiency. To this end, the recognition of worker activity allows for accurate quantification of performance metrics, improving efficiency holistically while contributing to worker safety. OpenMarcie is, to the best of our knowledge, the biggest multimodal dataset designed for human action monitoring in manufacturing environments. It includes data from wearables sensing modalities and cameras distributed in the surroundings. The dataset is structured around two experimental settings, involving a total of 36 participants. In the first setting, twelve participants perform a bicycle assembly and disassembly task under semi-realistic conditions without a fixed protocol, promoting divergent and goal-oriented problem-solving. The second experiment involves twenty-five volunteers (24 valid data) engaged in a 3D printer assembly task, with the 3D printer manufacturer's instructions provided to guide the volunteers in acquiring procedural knowledge. This setting also includes sequential collaborative assembly, where participants assess and correct each other's progress, reflecting real-world manufacturing dynamics. OpenMarcie includes over 37 hours of egocentric and exocentric, multimodal, and multipositional data, featuring eight distinct data types and more than 200 independent information channels. The dataset is benchmarked across three human activity recognition tasks: activity classification, open vocabulary captioning, and cross-modal alignment.