# RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models

arXiv: https://arxiv.org/abs/2407.12730 · HTML: https://mlsanity.com/paper/2407.12730/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 17 of 17)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning](https://mlsanity.com/paper/2312.12379/) (2023)
- [LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs](https://mlsanity.com/paper/2401.16160/) (2024)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food](https://mlsanity.com/paper/2103.03375/) (2021)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [FIRE: Food Image to REcipe generation](https://mlsanity.com/paper/2308.14391/) (2023)
- [Multi-modal Cooking Workflow Construction for Food Recipes](https://mlsanity.com/paper/2008.09151/) (2020)
- [Mining Discriminative Food Regions for Accurate Food Recognition](https://mlsanity.com/paper/2207.03692/) (2022)
- [From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios](https://mlsanity.com/paper/2403.07403/) (2024)
- [Synthesizing Knowledge-enhanced Features for Real-world Zero-shot Food Detection](https://mlsanity.com/paper/2402.09242/) (2024)

## Built on by (most cited first, 11 of 11)

- [EventHallusion: Diagnosing Event Hallucinations in Video LLMs](https://mlsanity.com/paper/2409.16597/) (2024)
- [ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing](https://mlsanity.com/paper/2412.14711/) (2024)
- [Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion](https://mlsanity.com/paper/2505.08747/) (2025)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning](https://mlsanity.com/paper/2411.12787/) (2024)
- [SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights](https://mlsanity.com/paper/2507.04412/) (2025)
- [OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion](https://mlsanity.com/paper/2604.12356/) (2026)
- [Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning](https://mlsanity.com/paper/2511.13351/) (2025)
- [FoodLogAthl-218: Constructing a Real-World Food Image Dataset Using Dietary Management Applications](https://mlsanity.com/paper/2512.14574/) (2025)
- [Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning](https://mlsanity.com/paper/2606.04986/) (2026)
- [ReGraph: Learning to Generate Recipe Graphs from Food Images](https://mlsanity.com/paper/2608.06917/) (2026)

## Abstract (arXiv)

Large Multi-modal Models (LMMs) have significantly advanced a variety of vision-language tasks. The scalability and availability of high-quality training data play a pivotal role in the success of LMMs. In the realm of food, while comprehensive food datasets such as Recipe1M offer an abundance of ingredient and recipe information, they often fall short of providing ample data for nutritional analysis. The Recipe1M+ dataset, despite offering a subset for nutritional evaluation, is limited in the scale and accuracy of nutrition information. To bridge this gap, we introduce Uni-Food, a unified food dataset that comprises over 100,000 images with various food labels, including categories, ingredients, recipes, and ingredient-level nutritional information. Uni-Food is designed to provide a more holistic approach to food data analysis, thereby enhancing the performance and capabilities of LMMs in this domain. To mitigate the conflicts arising from multi-task supervision during fine-tuning of LMMs, we introduce a novel Linear Rectification Mixture of Diverse Experts (RoDE) approach. RoDE utilizes a diverse array of experts to address tasks of varying complexity, thereby facilitating the coordination of trainable parameters, i.e., it allocates more parameters for more complex tasks and, conversely, fewer parameters for simpler tasks. RoDE implements linear rectification union to refine the router's functionality, thereby enhancing the efficiency of sparse task allocation. These design choices endow RoDE with features that ensure GPU memory efficiency and ease of optimization. Our experimental results validate the effectiveness of our proposed approach in addressing the inherent challenges of food-related multitasking.