mlsanityData to 14 Oct 2026

Accelerating Local AI on Consumer GPUs: A Hardware-Aware Dynamic Strategy for YOLOv10s

arXiv 2509.07928 ↗First posted 9 Sep 2025cs.CV cs.AI cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As local AI grows in popularity, there is a critical gap between the benchmark performance of object detectors and their practical viability on consumer-grade hardware. While models like YOLOv10s promise real-time speeds, these metrics are typically achieved on high-power, desktop-class GPUs. This paper reveals that on resource-constrained systems, such as laptops with RTX 4060 GPUs, performance is not compute-bound but is instead dominated by system-level bottlenecks, as illustrated by a simple bottleneck test. To overcome this hardware-level constraint, we introduce a Two-Pass Adaptive Inference algorithm, a model-independent approach that requires no architectural changes. This study mainly focuses on adaptive inference strategies and undertakes a comparative analysis of architectural early-exit and resolution-adaptive routing, highlighting their respective trade-offs within a unified evaluation framework. The system uses a fast, low-resolution pass and only escalates to a high-resolution model pass when detection confidence is low. On a 5000-image COCO dataset, our method achieves a 1.85x speedup over a PyTorch Early-Exit baseline, with a modest mAP loss of 5.51%. This work provides a practical and reproducible blueprint for deploying high-performance, real-time AI on consumer-grade devices by shifting the focus from pure model optimization to hardware-aware inference strategies that maximize throughput.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 16 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

10 of 16 references.

References · 10

  1. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  2. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 533
  3. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 515
  4. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  5. 2021Dynamic Neural Networks: A Surveycited by 408
  6. 2023A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAScited by 151
  7. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  8. 2022PP-YOLOE: An evolved version of YOLOcited by 75
  9. 2017Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectorscited by 12
  10. 2024YOLOv5 vs. YOLOv8 in Marine Fisheries: Balancing Class Detection and Instance Countcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines
  2. 2026Hardware Utilization and Inference Performance of Edge Object Detection Under Fault Injection
  3. 2025ZettaLith: An Architectural Exploration of Extreme-Scale AI Inference Acceleration
  4. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
  5. 2026On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend
  6. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  7. 2025Hardware optimization on Android for inference of AI models
  8. 2026A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi