关键词: 泊位岸桥调度, 深度强化学习, 连续泊位分配, 岸桥动态调度, PPO算法

Abstract: Container ports are important maritime hubs that connect trade between countries. In daily operations, ports need to allocate limited berth and quay crane resources to ships within a planning period based on multiple factors such as arrival time, ship type, workload, and departure plans, to ensure that all ships complete their operations as soon as possible. The decisions of berth allocation and quay crane scheduling usually rely on an intuition and experience of port staff, which can easily lead to a prolonged vessel stay in port or a waste of limited resources. Moreover, in the trend of large-sized ships, increasing port throughput, and the gradual popularization of intelligent equipment and systems, a large amount of operational data has been accumulated during daily operations. More scientific and intelligent decision-making methods are urgently required for berth and crane scheduling to further improve port operation efficiency and resources utilization, especially in complex large-scale dynamic environments.
The problem of port resources allocation and equipment scheduling has attracted the attention of many scholars, but existing research on berth allocation usually considers the coastline to be discrete into multiple berths, which cannot truly depict the actual situation of the coastline. And existing research on crane scheduling mostly considers static task sets and fails to meet the real-time scheduling needs of ports in complex environments. At present, many scholars have proposed various solutions to berth allocation and crane scheduling problems. But most of them consider the two related decisions separately, and ones using machine learning techniques only optimize the parameters in the traditional algorithms without taking full advantage of the adaptive and efficient characteristics of reinforcement learning.
In order to meet the needs of intelligent decision-making of port berth and crane scheduling in a large-scale complex dynamic environment, this paper considers a joint optimization problem of continuous berth allocation and quay crane dynamic scheduling. The problem is considered to be a sequencing one and is described as a Markov Decision Process (MDP) with carefully designed state space, action space, and reward functions. An efficient reinforcement learning method is proposed by combining A2C (Actor to Critical) neural network with Proximal Policy Optimization (PPO).
The PPO-based method is trained and tested based on real data from a port in Shanghai, China. Each episode of the training process contains 72 steps of interactions with the environment, and the agent updates the model every 100 steps. During the training process, the stage training results are saved every 1000 episodes. The training process stabilizes and converges after 60 thousand episodes. Based on test datasets of different scales, the proposed method can quickly generate optimal decisions for continuous berth allocation and quay crane dynamic scheduling. To show the performance of the proposed method, comparative experiments with the DDPG algorithm and three classic heuristics are conducted. Compared with DDPG, the proposed method can reduce the total working time of the ships by 3 to 10 hours, and has a better performance and efficiency of convergence in the training process. For 30 ships, compared with the genetic algorithm, PSO algorithm, and a first-come-first-serve heuristics, the proposed method reduces the total working time optimization objective by 15.7%, 20.3%, and 11%, respectively, and reduces decision time by about 93% compared to the two metaheuristics.
The PPO-based method proposed in this article can fully utilize historical data, obtain and update the best strategy through interactive training in a dynamic environment, and make effective decisions for continuous berth allocation and quay crane dynamic scheduling. It has more advantages in decision-making efficiency and optimization on objectives compared to traditional learning methods and is thus more in line with the current intelligent development requirements of ports. Future research could consider using reinforcement learning in multi-level equipment joint scheduling, and cooperative game of multiple agents.

Key words: berth and quay crane joint scheduling, deep reinforcement learning, continuous berth allocation, crane dynamic scheduling, PPO algorithm