# Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges

arXiv: https://arxiv.org/abs/2407.00092 · HTML: https://mlsanity.com/paper/2407.00092/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Learning Distilled Collaboration Graph for Multi-Agent Perception](https://mlsanity.com/paper/2111.00643/) (2021)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [Learning the Travelling Salesperson Problem Requires Rethinking Generalization](https://mlsanity.com/paper/2006.07054/) (2020)
- [Graph Neural Network Guided Local Search for the Traveling Salesperson Problem](https://mlsanity.com/paper/2110.05291/) (2021)
- [Boosted Prompt Ensembles for Large Language Models](https://mlsanity.com/paper/2304.05970/) (2023)
- [Multi-Vehicle Routing Problems with Soft Time Windows: A Multi-Agent Reinforcement Learning Approach](https://mlsanity.com/paper/2002.05513/) (2020)
- [Generalization of Machine Learning for Problem Reduction: A Case Study on Travelling Salesman Problems](https://mlsanity.com/paper/2005.05847/) (2020)
- [A Comprehensive Survey on the Multiple Travelling Salesman Problem: Applications, Approaches and Taxonomy](https://mlsanity.com/paper/2102.12772/) (2021)
- [Reinforced Lin-Kernighan-Helsgaun Algorithms for the Traveling Salesman Problems](https://mlsanity.com/paper/2207.03876/) (2022)
- [How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems](https://mlsanity.com/paper/2403.01757/) (2024)
- [Exploring Combinatorial Problem Solving with Large Language Models: A Case Study on the Travelling Salesman Problem Using GPT-3.5 Turbo](https://mlsanity.com/paper/2405.01997/) (2024)
- [Eyeballing Combinatorial Problems: A Case Study of Using Multimodal Large Language Models to Solve Traveling Salesman Problems](https://mlsanity.com/paper/2406.06865/) (2024)
- [Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events](https://mlsanity.com/paper/2406.13894/) (2024)
- [Exploring Traffic Crash Narratives in Jordan Using Text Mining Analytics](https://mlsanity.com/paper/2406.09438/) (2024)
- [Object Detection using Oriented Window Learning Vi-sion Transformer: Roadway Assets Recognition](https://mlsanity.com/paper/2406.10712/) (2024)
- [Random-Key Cuckoo Search for the Travelling Salesman Problem](https://mlsanity.com/paper/1607.04324/) (2016)
- [The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications](https://mlsanity.com/paper/2406.13898/) (2024)
- [Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.14152/) (2022)

## Built on by (most cited first, 21 of 21)

- [GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning](https://mlsanity.com/paper/2505.23399/) (2025)
- [Large Language Models for Combinatorial Optimization: A Systematic Review](https://mlsanity.com/paper/2507.03637/) (2025)
- [Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm](https://mlsanity.com/paper/2411.10869/) (2024)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing](https://mlsanity.com/paper/2409.18286/) (2024)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [Dual Latent Memory for Visual Multi-agent System](https://mlsanity.com/paper/2602.00471/) (2026)
- [GraphThought: Graph Combinatorial Optimization with Thought Generation](https://mlsanity.com/paper/2502.11607/) (2025)
- [Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies](https://mlsanity.com/paper/2508.05433/) (2025)
- [FinVision: A Multi-Agent Framework for Stock Market Prediction](https://mlsanity.com/paper/2411.08899/) (2024)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Bridging Synthetic and Real Routing Problems via LLM-Guided Instance Generation and Progressive Adaptation](https://mlsanity.com/paper/2511.10233/) (2025)
- [DRAGON: LLM-Driven Decomposition and Reconstruction Agents for Large-Scale Combinatorial Optimization](https://mlsanity.com/paper/2601.06502/) (2026)
- [HazardNet: A Small-Scale Vision Language Model for Real-Time Traffic Safety Detection at Edge Devices](https://mlsanity.com/paper/2502.20572/) (2025)
- [Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends](https://mlsanity.com/paper/2504.16134/) (2025)
- [ViTSP: A Vision Language Models Guided Framework for Solving Large-Scale Traveling Salesman Problems](https://mlsanity.com/paper/2509.23465/) (2025)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach](https://mlsanity.com/paper/2503.04918/) (2025)
- [Graph Optimization Foundation Model: Tokenizing Graph via A Language-Model Paradigm](https://mlsanity.com/paper/2509.24256/) (2025)
- [Constraint-aware Path Planning from Natural Language Instructions Using Large Language Models](https://mlsanity.com/paper/2603.19257/) (2026)
- [LM-GRASP: Instance-Specific Language Models for Combinatorial Construction via Online Imitation Learning](https://mlsanity.com/paper/2607.28135/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) harness comprehensive knowledge spanning text, images, and audio to adeptly tackle complex problems, including zero-shot in-context learning scenarios. This study explores the ability of MLLMs in visually solving the Traveling Salesman Problem (TSP) and Multiple Traveling Salesman Problem (mTSP) using images that portray point distributions on a two-dimensional plane. We introduce a novel approach employing multiple specialized agents within the MLLM framework, each dedicated to optimizing solutions for these combinatorial challenges. Our experimental investigation includes rigorous evaluations across zero-shot settings and introduces innovative multi-agent zero-shot in-context scenarios. The results demonstrated that both multi-agent models. Multi-Agent 1, which includes the Initializer, Critic, and Scorer agents, and Multi-Agent 2, which comprises only the Initializer and Critic agents; significantly improved solution quality for TSP and mTSP problems. Multi-Agent 1 excelled in environments requiring detailed route refinement and evaluation, providing a robust framework for sophisticated optimizations. In contrast, Multi-Agent 2, focusing on iterative refinements by the Initializer and Critic, proved effective for rapid decision-making scenarios. These experiments yield promising outcomes, showcasing the robust visual reasoning capabilities of MLLMs in addressing diverse combinatorial problems. The findings underscore the potential of MLLMs as powerful tools in computational optimization, offering insights that could inspire further advancements in this promising field. Project link: https://github.com/ahmed-abdulhuy/Solving-TSP-and-mTSP-Combinatorial-Challenges-using-Visual-Reasoning-and-Multi-Agent-Approach-MLLMs-.git