mlsanityData to 7 Oct 2026

Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors

arXiv 2408.08302 ↗First posted 15 Aug 2024cs.AI cs.CL cs.LGcited by 13

On the map: LLM reasoning → Chain-of-thought reasoning for driving

Abstract (arXiv)

In this paper, we explore the capabilities of state-of-the-art large language models (LLMs) such as GPT-4, GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3, and Llama 3.1 in solving some selected undergraduate-level transportation engineering problems. We introduce TransportBench, a benchmark dataset that includes a sample of transportation engineering problems on a wide range of subjects in the context of planning, design, management, and control of transportation systems. This dataset is used by human experts to evaluate the capabilities of various commercial and open-sourced LLMs, especially their accuracy, consistency, and reasoning behaviors, in solving transportation engineering problems. Our comprehensive analysis uncovers the unique strengths and limitations of each LLM, e.g. our analysis shows the impressive accuracy and some unexpected inconsistent behaviors of Claude 3.5 Sonnet in solving TransportBench problems. Our study marks a thrilling first step toward harnessing artificial general intelligence for complex transportation challenges.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 28 of its 57 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

28 of 57 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2022Large Language Models Can Self-Improvecited by 656
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
16 more
  1. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  2. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  3. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  4. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  5. 2023A Survey of Reasoning with Foundation Modelscited by 70
  6. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 67
  7. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  8. 2023Dynamic Planning with a LLMcited by 60
  9. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  10. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  11. 2023TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safetycited by 27
  12. 2023Structured Chemistry Reasoning with Large Language Modelscited by 27
  13. 2023Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4cited by 16
  14. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 15
  15. 2023ChatGPT is on the Horizon: Could a Large Language Model be Suitable for Intelligent Traffic Safety Research and Applications?cited by 9
  16. 2024Unlocking the Potential: Benchmarking Large Language Models in Water Engineering and Researchcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  2. 2024ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertisecited by 20
  3. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 14
  4. 2026Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimizationcited by 13
  5. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 10
  6. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  7. 2024Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Textcited by 7
  8. 2024TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Modelscited by 5
  9. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoningcited by 4
  10. 2024Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generationcited by 2
  11. 2025CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenancecited by 1
  12. 2026SupChain-Bench: Benchmarking Large Language Models for Real-World Supply Chain Managementcited by 0
1 more
  1. 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Supportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models
  2. 2026TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
  3. 2025LLM-ABM for Transportation: Assessing the Potential of LLM Agents in System Analysis
  4. 2026Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline
  5. 2024Beyond Words: Evaluating Large Language Models in Transportation Planning
  6. 2024Multimodal LLM for Intelligent Transportation Systems
  7. 2024Multimodal Road Network Generation Based on Large Language Model ↗
  8. 2026MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes