# Multi-Modal Fusion of In-Situ Video Data and Process Parameters for Online Forecasting of Cookie Drying Readiness

arXiv: https://arxiv.org/abs/2504.15599 · HTML: https://mlsanity.com/paper/2504.15599/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 62 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Federated Transfer Learning with Task Personalization for Condition Monitoring in Ultrasonic Metal Welding](https://mlsanity.com/paper/2404.13278/) (2024)
- [Multi-Modal Data Fusion for Moisture Content Prediction in Apple Drying](https://mlsanity.com/paper/2504.07465/) (2025)
- [WeldMon: A Cost-effective Ultrasonic Welding Machine Condition Monitoring System](https://mlsanity.com/paper/2308.05756/) (2023)
- [Reinforcement Learning Constrained Beam Search for Parameter Optimization of Paper Drying Under Flexible Constraints](https://mlsanity.com/paper/2501.12542/) (2025)

## Built on by (most cited first, 3 of 3)

- [XPG-RL: Reinforcement Learning with Explainable Priority Guidance for Efficiency-Boosted Mechanical Search](https://mlsanity.com/paper/2504.20969/) (2025)
- [Certifiably Safe Manipulation of Deformable Linear Objects via Joint Shape and Tension Prediction](https://mlsanity.com/paper/2505.13889/) (2025)
- [Multi-Modal Zero-Shot Prediction of Color Trajectories in Food Drying](https://mlsanity.com/paper/2512.06190/) (2025)

## Abstract (arXiv)

Food drying is essential for food production, extending shelf life, and reducing transportation costs. Accurate real-time forecasting of drying readiness is crucial for minimizing energy consumption, improving productivity, and ensuring product quality. However, this remains challenging due to the dynamic nature of drying, limited data availability, and the lack of effective predictive analytical methods. To address this gap, we propose an end-to-end multi-modal data fusion framework that integrates in-situ video data with process parameters for real-time food drying readiness forecasting. Our approach leverages a new encoder-decoder architecture with modality-specific encoders and a transformer-based decoder to effectively extract features while preserving the unique structure of each modality. We apply our approach to sugar cookie drying, where time-to-ready is predicted at each timestamp. Experimental results demonstrate that our model achieves an average prediction error of only 15 seconds, outperforming state-of-the-art data fusion methods by 65.69% and a video-only model by 11.30%. Additionally, our model balances prediction accuracy, model size, and computational efficiency, making it well-suited for heterogenous industrial datasets. The proposed model is extensible to various other industrial modality fusion tasks for online decision-making.