关键词: 分布鲁棒优化, 机器学习, Gelbrich不确定集, 线性回归

Abstract: Stochastic optimization can effectively describe the decision-making related problems with uncertain factors. One of the key problems in stochastic optimization research is to determine the distribution information of random variables. With the increasing complexity of practical problems, it becomes more and more difficult to obtain the distribution information of random variables accurately. In view of the situation that the distribution information of random variables is incomplete, optimization scholars have developed stochastic optimization theories and methods, and proposed the distributionally robust optimization, which is widely used in decision-making related problems in practical fields. The distributionally robust optimization combines the traditional robust optimization and stochastic optimization methods, which can effectively deal with the optimal decision-making of random variables under uncertain probability distribution. The key issues of distributionally robust optimization include the construction of uncertain distribution sets and the transformation of models. The ambiguity sets based on Wasserstein metric have been widely studied. However, as soon as one of the two distributions is no longer discrete, the Wasserstein distance cannot be computed in polynomial time, that is, computing the Wasserstein distance is generally a #P-hard problem. Therefore, new measures need to be developed. On the other hand, the distributionally robust optimization is widely used in machine learning. For example, the distributionally robust optimization can be used for outlier detection and processing, and can also improve the classification accuracy of the model and so on.
Although the application of distributionally robust optimization in machine learning has great potential, there are still many challenges and problems to be solved. One is how to effectively construct ambiguity sets, and the other is how to convert the distributionally robust optimization model into a solvable problem. Based on Gelbrich ambiguity sets, this paper constructs a distributionally robust optimization model, and transforms the model into a form that is easy to calculate and solve by optimization duality theory. Then the model is applied to the linear regression problem of machine learning. Under certain assumptions, the previous conclusion is applied to prove that the model is equivalent to a semidefinite programming problem. Finally, we select the “red win” data set from UCI machine learning repository, which is commonly used for regression analysis, for numerical experiments. The goal of the data set is to predict the quality score of red wine, which contains 1599 samples. Each data point has 11 features (such as fixed acidity, remaining sugar, alcohol, etc.) and a label (quality score). We first normalize each feature and label of the dataset by min-max to eliminate dimensional differences between features. Then the effectiveness of the model is verified from three aspects: the influence of the change of radius on the optimal value of error, the influence of the change of sample size on the optimal value of error, and the comparison between Gelbrich ambiguity sets and Wasserstein ambiguity sets.
From the numerical experiments, we obtain the following conclusions: (1)Theoretically, with an increase in the radius, the optimal value of the distributionally robust interior problem becomes larger, and thus the optimal value of the whole distributionally robust optimization becomes larger. The actual calculation results verify the theory. (2)Taking the same number of different random samples and repeating ten times to obtain the ten optimal values of the same problem, it can be obtained that with an increase in the sample size, the Gelbrich distributionally robust minimum absolute error tends to be the same. This indicates that the learned model becomes more and more stable as the sample size continues to increase. (3)The Gelbrich distributionally robust least absolute error model is compared with the type-1 Wasserstein distributionally robust least absolute error model. It is found that the error of the former is smaller than that of the latter, that is, GDR-LAE has a better fitting effect on the data, which further verifies the calculation effect of the proposed model.
Since this paper only considers the linear regression problem in machine learning, how to construct and solve the distributionally robust optimization model for nonlinear regression problems and optimization problems such as classification in machine learning can be further studied.

Key words: distributionally robust optimization, machine learning, Gelbrich ambiguity set, linear regression