关键词: 深度强化学习, 投资组合模型, 变分模态分解, 多尺度特征提取

Abstract: Portfolio selection is the process of allocating wealth dynamically among a group of assets with the goal of maximizing long-term returns with a certain level of risk tolerance or minimizing risk with a certain expected return on the portfolio. With the development of computer technology, constructing a portfolio selection model based on machine learning has become a hot spot in current intelligent financial investment research. But in traditional machine learning and deep learning, supervised learning is usually used to predict various asset prices, and cannot directly interact with financial market, which is still defective in constructing portfolio selection models. Firstly, deep learning feature extraction focuses more on short-term returns at the expense of long-term returns, which creates more risk in the portfolio. Secondly, deep learning models cannot dynamically adjust their trading strategies as the market changes.
However, unlike other machine learning methods, deep reinforcement learning is centered on the interaction of the agent with the environment. Therefore, with the goal of maximizing the reward function and learning from feedback signals, deep reinforcement learning is better suited for solving nonlinear problems with delayed returns such as portfolio selection. Existing deep reinforcement learning methods for solving portfolio problems can be broadly categorized into three types: value-based, strategy gradient-based, and actor-critic. Value-based algorithms suffer from high bias and are often used to solve problems in discrete spaces. Therefore, these algorithms are not suitable for solving portfolio problems with continuous action spaces. Policy gradient-based algorithms suffer from training instability and policy convergence difficulties due to their presence of high variance and noisy gradients. The algorithms based on the actor-critic framework combine the above two methods, and solve the contradiction between high bias and high variance. This kind of algorithm is able to generate strategies directly through the actor network as well as evaluate the goodness of the strategies in real time through the critic network, which are more suitable for solving portfolio problems. Proximal Policy Optimization (PPO) is an algorithm based on actor-critic framework as well as one of the SOTA algorithms in the field of reinforcement learning. Therefore, in this paper, PPO will be utilized as a framework to construct a stock portfolio model. Furthermore, there is a lot of short-term speculation and noisy trading in the capital market, which makes the financial time series data contain a lot of noise. Specifically, in the short term, the price of assets will fluctuate irregularly due to a large number of short-term speculative and noise trading; however, in the long term, the price of assets will return to its value due to the law of value. Therefore, the high-frequency fluctuations of asset prices contain more noise; correspondingly, their low-frequency fluctuations contain more valid information.
Aiming at above problems, this paper proposes a deep reinforcement learning portfolio model VMD-PPO. Firstly, this model decomposes the stock price time series using the VMD based on the SSA algorithm determining the parameters to obtain k intrinsic mode functions (IMF) with different center frequency. Secondly, it removes some high-frequency IMFs in the decomposition results and reconstructs the decomposition stock price time series using the gray correlation clustering method to obtain the high and low frequency terms and trend terms. This step can reduce noise in financial time-series data. Thirdly, the volatility feature extraction network is constructed to learn the multiscale features of the stock price time series. Finally, the optimal portfolio model and the corresponding portfolio strategy based on PPO algorithm are constructed.
To verify the validity of our model, 20 constituent stocks in 10 industries from CSI300 andCSI500 are randomly selected for back testing separately. Four indicators, including cumulative return, Sharpe ratio, maximum retracement and Karma ratio are used as assessment indicators. Multiple experimental results show that VMD-PPO can effectively reduce the noise of financial time-series data and efficiently extract the multi-timescale features therein, which significantly outperforms the other control group models, and can better control the risk in different market environments and obtain more excess returns.

Key words: deep reinforcement learning, portfolio model, VMD, multi-scale feature extraction