Abstract: The marking problems of large-scale subjective-type competitions or examinations belong to typical group decision-making problems, due to the large number of participants and limited number of experts and marking time, and each answer sheet can only be randomly assigned to a few experts for marking, so the scoring matrices of large-scale competitions are incomplete. Subjective questions generally do not have standard answers, and are susceptible to the subjective factors of experts during the marking process, which can produce scoring system errors. Scoring system errors can be divided into two categories. The first is unequal mean scores, that is, some experts score generally higher, while others score generally lower. The second is unequal variance, which means that some experts’ ratings are more varied (large variance), while others are less varied (small variance). Due to the scoring system error, the raw scores of different judges are not additive in the incomplete scoring, so the traditional scoring method, that is, taking the average of the raw scores directly, is unfair to the contestants. At present, the T-score method (standardized score method) is often used, i.e., the mean score and variance of the answer sheets reviewed by each expert are leveled to the same level. However, in incomplete scoring, each expert reviews different answer sheets and the mean level of each expert’s answer sheet is different, so in incomplete scoring, the direct use of T-score method is not scientific enough.
In order to reduce the systematic error in scoring between experts, firstly we define the concept of “feature information” and its calculation formula, which reflects, to a certain extent, the degree of leniency or preference of an expert’s evaluation for the participant and represents the expert’s evaluation characteristics. Secondly, inspired by the formula of information entropy, we define the concept of “correlation information” and its calculation formula, and establish the least sum of square error model for determining expert weights based on the pairwise comparison matrix of correlation information. In order to test the reliability of the method of determining expert weights proposed in this paper, we conduct 100 simulation experiments with the example of Undergraduate Mathematical Contest in Modeling. The simulation experiments are divided into two groups: the number of modeling papers in the first group of experiments is 40 and the number of judges is 5; the number of modeling papers in the second group of experiments is 100 and the number of judges is 8. Each paper is reviewed by three experts, and each group of experiments is simulated for 50 times. When assigning papers to experts, we follow the principles of even and cross-assignment in order to increase the comparability of ratings among experts and make the pairwise comparison matrix of correlation information more reliable. In order to illustrate the scientificity of the method in this paper, in the simulation experiments, we use the traditional method (directly taking the mean value of the original score), the T-score method (standardized score method) and the method of this paper for comparative analysis respectively, and use the consistency rate, difference degree, error degree and controversy degree as the indexes for evaluating the advantages and disadvantages of the methods. The results of simulation experiments show that our method has higher consistency rate and lower difference, error and controversy degrees than the traditional method and T-score method, which indicates that our method is more scientific and reasonable than the other two methods.
The method of determining experts’ weights in this paper fully considers systematic errors, but less consideration is given to factors such as random errors and scoring drift. Therefore, further research can add random error, scoring drift and other factors to design a dynamic expert weight. The simulation experiments in this paper are based on the comprehensive scoring method, and further research can consider the itemized scoring method, that is, in the case of the itemized scoring method, how the posteriori weights are designed to improve the quality of the expert marking and reduce the system error.

Key words: group decision-making, expert’s weight, systematic error, subjective scoring, correlation information

摘要： 专家权重在群决策中具有重要的作用,影响着评价与决策的质量。针对残缺主观评分型群决策问题,为了提高评判的准确性和公平性,减小决策偏差,提出了一种充分考虑评分系统误差的专家权重计算方法。首先,定义了专家评价的特征信息和关联信息;其次,根据专家评价的关联信息构造成对比较矩阵,并应用误差平方和最小法确定专家的后验权重;最后,以全国大学生数学建模竞赛为例,进行了两组不同规模的100次仿真模拟实验,并通过一致率、差异度、误差度和争议度等检验指标进行比较分析。结果表明,本文方法较传统法和标准分法(T分数法)提高了一致率,减小了差异度、误差度和争议度,验证了该方法的有效性和科学性。

关键词: 群决策, 专家权重, 系统误差, 主观评分, 关联信息