mlsanityData to 14 Oct 2026

FineGen: A VLM-based Multi-Agent Framework for Fine-Grained Image-Text Dataset Construction

arXiv 2606.07645 ↗First posted 2 Jun 2026cs.CV cs.AIcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

The scarcity of hard negative samples in current vision-language datasets significantly hinders fine-grained perception. To address this, we propose FineGen, a VLM-based Multi-Agent framework for automated dataset construction. By employing a collaborative Generation-Verification-Correction pipeline with a closed-loop feedback mechanism, FineGen ensures synthesized hard negatives are semantically valid yet strictly contradictory to visual content. Applying this to ImageNet, we construct FineGen-100K, a hierarchical dataset containing over 147,000 attribute-specific hard negatives with a rigorous 1:10 positive-to-negative ratio. Extensive evaluations confirm a 96.7% attribute validity rate. Crucially, downstream validation on the FG-OVD benchmark shows that fine-tuning on FineGen-100K yields a substantial +14.4% accuracy improvement on hard samples, significantly outperforming state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 25 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

22 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation17 Dec
  2. 2025AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data7 May

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2020Contrastive Learning with Hard Negative Samplescited by 628
8 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  4. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  5. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  6. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  7. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understandingcited by 29
  8. 2024Is CLIP the main roadblock for fine-grained open-world perception?cited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FineVision: Open Data Is All You Need
  2. 2026M3: High-fidelity Text-to-Image Generation via Multi-Modal, Multi-Agent and Multi-Round Visual Reasoning
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use
  5. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  6. 2023ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data
  7. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  8. 2026PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensembles