# Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development

arXiv: https://arxiv.org/abs/2407.11784 · HTML: https://mlsanity.com/paper/2407.11784/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [Efficient Multimodal Learning from Data-centric Perspective](https://mlsanity.com/paper/2402.11530/) (2024)
- [EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagation](https://mlsanity.com/paper/2405.18991/) (2024)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)

## Built on by (most cited first, 10 of 10)

- [Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models](https://mlsanity.com/paper/2408.04594/) (2024)
- [Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data](https://mlsanity.com/paper/2502.04380/) (2025)
- [Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models](https://mlsanity.com/paper/2501.14755/) (2024)
- [MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?](https://mlsanity.com/paper/2503.09499/) (2025)
- [DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?](https://mlsanity.com/paper/2505.16915/) (2025)
- [A Survey of LLM × DATA](https://mlsanity.com/paper/2505.18458/) (2025)
- [SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discovery](https://mlsanity.com/paper/2602.09132/) (2026)
- [Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs](https://mlsanity.com/paper/2510.25441/) (2025)
- [DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning](https://mlsanity.com/paper/2602.11089/) (2026)
- [VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes](https://mlsanity.com/paper/2609.06652/) (2026)

## Abstract (arXiv)

The emergence of multimodal large models has advanced artificial intelligence, introducing unprecedented levels of performance and functionality. However, optimizing these models remains challenging due to historically isolated paths of model-centric and data-centric developments, leading to suboptimal outcomes and inefficient resource utilization. In response, we present a new sandbox suite tailored for integrated data-model co-development. This sandbox provides a feedback-driven experimental platform, enabling cost-effective iteration and guided refinement of both data and models. Our proposed ``Probe-Analyze-Refine'' workflow, validated through practical use cases on multimodal tasks such as image-text pre-training with CLIP, image-to-text generation with LLaVA-like models, and text-to-video generation with DiT-based models, yields transferable and notable performance boosts, such as topping the VBench leaderboard. A comprehensive set of over 100 experiments demonstrated the suite's usability and extensibility, while also uncovering insights into the interplay between data quality, diversity, model behavior, and computational costs. All codes, datasets, and models are open-sourced to foster future research and applications that would otherwise be infeasible due to the lack of a dedicated co-development infrastructure.