mlsanityData to 14 Oct 2026

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation

arXiv 2311.13258 ↗First posted 22 Nov 2023cs.CV cs.CL cs.LGcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, we present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in a well-organized structured format. Second, we introduce curriculum-based learning for VLMs to progressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute to complex visual structure understanding. Furthermore, we compile and release a collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web. In experiments, we evaluate ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code is public at urlhttps://github.com/Yangyi-Chen/vi-struct.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 92 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

57 of 92 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  2. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  6. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  7. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
38 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  4. 2021ImageNet-21K Pretraining for the Massescited by 613
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  7. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  8. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  9. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  11. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2017Automated Curriculum Learning for Neural Networkscited by 257
  14. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  15. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  16. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  17. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  18. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  19. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  20. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 107
  21. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
  22. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  23. 2020Grounded Situation Recognitioncited by 78
  24. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  25. 2020Cross-media Structured Common Space for Multimedia Event Extractioncited by 61
  26. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  27. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  28. 2023OvarNet: Towards Open-vocabulary Object Attribute Recognitioncited by 43
  29. 2017Situation Recognition with Graph Neural Networkscited by 34
  30. 2019Scene Graph Prediction with Limited Labelscited by 32
  31. 2022Open-vocabulary Attribute Detectioncited by 29
  32. 2020Dialogue Response Selection with Hierarchical Curriculum Learningcited by 27
  33. 2021Visual Distant Supervision for Scene Graph Generationcited by 25
  34. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  35. 2022Collaborative Transformers for Grounded Situation Recognitioncited by 19
  36. 2016Commonly Uncommon: Semantic Sparsity in Situation Recognitioncited by 13
  37. 2022Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxiescited by 13
  38. 2023Knowledge-augmented Few-shot Visual Relation Detectioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  2. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  3. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  4. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 41
  5. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  6. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 15
  7. 2024CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Promptscited by 10
  8. 2024CodeGraph: Enhancing Graph Reasoning of LLMs with Codecited by 6
  9. 2025Event Extraction in Large Language Modelcited by 5
  10. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 2
  11. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
  12. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1
1 more
  1. 2026MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Models
  2. 2024Beyond Embeddings: The Promise of Visual Table in Visual Reasoning
  3. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
  4. 20233VL: Using Trees to Improve Vision-Language Models' Interpretability
  5. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  6. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations
  7. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  8. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs