Symbolic Representation for Any-to-Any Generative Tasks
Not on the map. Why?
Abstract (arXiv)
We propose a symbolic generative task description language and a corresponding inference engine capable of representing arbitrary multimodal tasks as structured symbolic flows. Unlike conventional generative models that rely on large-scale training and implicit neural representations to learn cross-modal mappings, often at high computational cost and with limited flexibility, our framework introduces an explicit symbolic representation comprising three core primitives: functions, parameters, and topological logic. Leveraging a pre-trained language model, our inference engine maps natural language instructions directly to symbolic workflows in a training-free manner. Our framework successfully performs over 12 diverse multimodal generative tasks, demonstrating strong performance and flexibility without the need for task-specific tuning. Experiments show that our method not only matches or outperforms existing state-of-the-art unified models in content quality, but also offers greater efficiency, editability, and interruptibility. We believe that symbolic task representations provide a cost-effective and extensible foundation for advancing the capabilities of generative AI.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 24 Apr 2025.
Builds on
28 of 45 references.
References · 28
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
- 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
16 more
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
- 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
- 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
- 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
- 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
- 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
- 2023Any-to-Any Generation via Composable Diffusioncited by 204
- 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
- 2017Inferring and Executing Programs for Visual Reasoningcited by 130
- 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
- 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systemscited by 30
- 2021Video-Text Pre-training with Learned Regionscited by 20
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.