mlsanityData to 14 Oct 2026

Concept-Oriented Deep Learning with Large Language Models

arXiv 2306.17089 ↗First posted 29 Jun 2023cs.LG cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Language Models (LLMs) have been successfully used in many natural-language tasks and applications including text generation and AI chatbots. They also are a promising new technology for concept-oriented deep learning (CODL). However, the prerequisite is that LLMs understand concepts and ensure conceptual consistency. We discuss these in this paper, as well as major uses of LLMs for CODL including concept extraction from text, concept graph extraction from text, and concept learning. Human knowledge consists of both symbolic (conceptual) knowledge and embodied (sensory) knowledge. Text-only LLMs, however, can represent only symbolic (conceptual) knowledge. Multimodal LLMs, on the other hand, are capable of representing the full range (conceptual and sensory) of human knowledge. We discuss conceptual understanding in visual-language LLMs, the most important multimodal LLMs, and major uses of them for CODL including concept extraction from image, concept graph extraction from image, and concept learning. While uses of LLMs for CODL are valuable standalone, they are particularly valuable as part of LLM applications such as AI chatbots.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 18 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

11 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial26 Jun
  2. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 9

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  3. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  4. 2023Human-like Few-Shot Learning via Bayesian Reasoning over Natural Languagecited by 22
  5. 2022Unpacking Large Language Models with Conceptual Consistencycited by 18
  6. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  7. 2021CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domaincited by 7
  8. 2018Concept-Oriented Deep Learningcited by 5
  9. 2022Dual Embodied-Symbolic Concept Representations for Deep Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 15

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  2. 2024A Review of Multi-Modal Large Language and Vision Models
  3. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
  4. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2023A Comprehensive Overview of Large Language Models
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  8. 2026Understanding Large Language Models