Improving Q-Learning for Real-World Control: A Case Study in Series Hybrid Agricultural Tractors

The variable and unpredictable load demands in hybrid agricultural tractors make it difficult to design optimal rule-based energy management strategies, motivating the use of adaptive, learning-based control. However, existing approaches often rely on basic fuel-based rewards and do not leverage expert demonstrations to accelerate training. In this paper, first, the performance of Q-value-based reinforcement learning algorithms is evaluated for powertrain control in a hybrid agricultural tractor. Three algorithms, Double Q-Learning (DQL), Deep Q-Networks (DQN), and Double DQN (DDQN), are compared in terms of convergence speed and policy optimality. Second, a piecewise domain-specific reward-shaping strategy is introduced to improve learning efficiency and steer agent behavior toward engine fuel-efficient operating regions. Third, the design of the experience replay buffer is examined, with a focus on the effects of seeding the buffer with expert demonstrations and analyzing how different types of expert policies influence convergence dynamics and final performance. Experimental results demonstrate that (1) DDQN achieves 70% faster convergence than DQN in this application domain, (2) the proposed reward shaping method effectively biases the learned policy toward fuel-efficient outcomes, and (3) initializing the replay buffer with structured expert data leads to a 33% improvement in convergence speed.