《电子商务 E-business》阅读文献：A decision support approach for assigning reviewers to proposals

Expert Systems with Applications 37(2010)6948-6956 Contents lists available at Science Direct Expert Systems with Applications ELSEVIER journalhomepagewww.elsevier.com/locate/eswa a decision support approach for assigning reviewers to proposals Yunhong Xua.b,, Jian Ma, Yonghong Sun, Gang Hao, Wei Xu Dingtao Zhao b PR China g Kong Kowloon, Hong Kong PR China d school of Management, Chinese Academy of Sciences. PR China ARTICLE IN FO A BSTRACT Peer review plays an nt role in research project selection at funding agencies. Quality of pe view ree of ma g This kind of matching is largely determined by the process of assigning proposals to reviewers. As the Research project selection umber of proposals submitted to funding agencies continues to grow, the traditional approaches of find- ng appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro- poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based the proposed approach to facilitate the decision making process of assigning proposals to reviewers. e 2010 Elsevier Ltd. All rights reserved. 1 Introduction Traditional assignment methods rely heavily on a single deci sion maker (e.g, panel chair) to manually analyze the title, ab- Selecting appropriate research project for funding is an impor- stract, keywords and other parts of the proposals and then tant task in government funding agencies(Tian, Ma, liu, 2002). identify a set of reviewers who are most likely to review the pro- Research project selection is also a complex one which often in- posals. Furthermore, constraints that should be considered when volves many sub-tasks (Cook, Golany, Kress, Penn, Raviv, assigning proposals to reviewers make the assignment problem 2005). It generally begins with a call for proposals(CFP)which de- more complicated. For example, proposals should be evaluated scribes funding opportunities and requirements of the funding by reviewers who are knowledgeable in the corresponding re- agencies, and then the CfP is distributed to relevant communities, search area, and relationships between applicants and reviewers such as universities and research institutions. Researchers in rele- that could affect the justice of review process (e.g, co-authorships) vant communities who are interested in CFP-related topics can need to be avoided. It presents a great challenge for managers who then submit their proposals to the corresponding funding agencies. are responsible for the assignment task, especially when there are Submitted proposals are validated, compiled and then assigned to large amounts of proposals and reviewers. field experts who are invited as reviewers to provide their opin Several studies have been carried out to solve the reviewer ions. These reviewers comment on the proposals based on their assignment problem from different perspectives, and various ap- orofessional knowledge and with reference to the specific criteria proaches have been proposed. These approaches can be classified issued by the funding agencies. The reviewing results are aggre into two major categories based on their underlying techniques: gated to determine which proposals would be funded (tian et al., approaches based on information retrieval, and approaches based 2002). Assigning proposals to reviewers is one of the most impor ion(Wang, Chen, Miao, 2008). The former ap- tant and challenging tasks(Sun, Ma, Fan, Wang, 2008). It must be proaches focus on using information retrieval techniques to com lone appropriately because assigning proper reviewers to a re- pute the matching degree between proposals and reviewers search proposal ensures reviewers have enough expertise to judge (david Andrew, 2007: Dumais Nielsen, 1992; Hettich Pazza- the quality of the proposal ni,2006: Rodriguez Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob lem from mathematical or operational research directions( Cook Technology of China, Hefei, Anhui Province, PR China. Tel. +86 512 87161393 et al, 2005: Janak, Taylor, Floudas, Burka, Mountziaris, 2006 fax:+8651287161381 Merelo-Guervos Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174 front matter o 2010 Elsevier Ltd. All rights reserved. oi:10.1016/eswa201003.027

A decision support approach for assigning reviewers to proposals Yunhong Xu a,b,*, Jian Ma a , Yonghong Sun a , Gang Hao c , Wei Xu d , Dingtao Zhao b aDepartment of Information Systems, City University of Hong Kong, Kowloon, Hong Kong, PR China b Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China cDepartment of Management Sciences, City University of Hong Kong, Kowloon, Hong Kong, PR China d School of Management, Chinese Academy of Sciences, Beijing, PR China article info Keywords: Proposal assignment Reviewer assignment Proposal grouping Research project selection abstract Peer review plays an important role in research project selection at funding agencies. Quality of peer review greatly depends on the degree of matching between reviewers and assigned research proposals. This kind of matching is largely determined by the process of assigning proposals to reviewers. As the number of proposals submitted to funding agencies continues to grow, the traditional approaches of find￾ing appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro￾poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based on the proposed approach to facilitate the decision making process of assigning proposals to reviewers. 2010 Elsevier Ltd. All rights reserved. 1. Introduction Selecting appropriate research project for funding is an impor￾tant task in government funding agencies (Tian, Ma, & Liu, 2002). Research project selection is also a complex one which often in￾volves many sub-tasks (Cook, Golany, Kress, Penn, & Raviv, 2005). It generally begins with a call for proposals (CFP) which de￾scribes funding opportunities and requirements of the funding agencies, and then the CFP is distributed to relevant communities, such as universities and research institutions. Researchers in rele￾vant communities who are interested in CFP-related topics can then submit their proposals to the corresponding funding agencies. Submitted proposals are validated, compiled and then assigned to field experts who are invited as reviewers to provide their opin￾ions. These reviewers comment on the proposals based on their professional knowledge and with reference to the specific criteria issued by the funding agencies. The reviewing results are aggre￾gated to determine which proposals would be funded (Tian et al., 2002). Assigning proposals to reviewers is one of the most impor￾tant and challenging tasks (Sun, Ma, Fan, & Wang, 2008). It must be done appropriately because assigning proper reviewers to a re￾search proposal ensures reviewers have enough expertise to judge the quality of the proposal. Traditional assignment methods rely heavily on a single deci￾sion maker (e.g., panel chair) to manually analyze the title, ab￾stract, keywords and other parts of the proposals and then identify a set of reviewers who are most likely to review the pro￾posals. Furthermore, constraints that should be considered when assigning proposals to reviewers make the assignment problem more complicated. For example, proposals should be evaluated by reviewers who are knowledgeable in the corresponding re￾search area, and relationships between applicants and reviewers that could affect the justice of review process (e.g., co-authorships) need to be avoided. It presents a great challenge for managers who are responsible for the assignment task, especially when there are large amounts of proposals and reviewers. Several studies have been carried out to solve the reviewer assignment problem from different perspectives, and various ap￾proaches have been proposed. These approaches can be classified into two major categories based on their underlying techniques: approaches based on information retrieval, and approaches based on optimization (Wang, Chen, & Miao, 2008). The former ap￾proaches focus on using information retrieval techniques to com￾pute the matching degree between proposals and reviewers (David & Andrew, 2007; Dumais & Nielsen, 1992; Hettich & Pazza￾ni, 2006; Rodriguez & Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob￾lem from mathematical or operational research directions (Cook et al., 2005; Janak, Taylor, Floudas, Burka, & Mountziaris, 2006; Merelo-Guervos & Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.03.027 * Corresponding author at: Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China. Tel.: +86 512 87161393; fax: +86 512 87161381. E-mail address: xuyunhong@mail.ustc.edu.cn (Y. Xu). Expert Systems with Applications 37 (2010) 6948–6956 Contents lists available at ScienceDirect Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 6949 assignment of candidate reviewers for each proposal, using either the aforementioned decision process, including the matching de- one or both of these approaches(Papagelis, Plexousakis, Niko- gree calculation model, the proposal grouping model and assignment model to assign reviewers to proposal groups, etc. According to our literature review, the majority of existing ap- The remainder of this paper is organized as follows. The re- proaches either from information retrieval stream or from optimi- search background is introduced in Section 2. Section 3 proposes ation stream are based on the same strategy which seeks an approach for solving the assignment problem. Section 4 pre earch appropriate reviewers for each individual proposal (as sents a prototype system based on the proposed approach. Section shown in Part A of Fig. 1). This strategy is in an attempt to ensure 5 validates the proposed approach and discusses the potential that every reviewer has sufficient expertise to evaluate the merits application in government funding agencies. Section 6 concludes of the proposals assigned to him/her. This strategy has several lim- the paper. itations: first, selecting appropriate reviewers for every individual proposal is time-consuming. For large funding agencies, the num ber of proposals received and the number of qualified reviewers 2. Backgro can be very large. Besides, the review process must be usually com- pleted with a tight deadline(Wang et al, 2008). Thus, partitioning 2.1. Background of NSFC proposals into groups and searching reviewers for each proposal up can be an efficient alternative. The time spent on reviewer one of the largest and most reputable research funding agen- assignment can be reduced by assigning reviewers to proposal cies in China, NSFC (National Natural Science Foundation of China) groups instead of individual proposals. Second, assigning individ- aims to fund research projects that have great potential of ual proposals to reviewers can hardly meet the specific require- tific and social impacts. NSFC has one general office, five bureaus, nents of the funding agencies, e.g. balancing the backgrounds of and seven scientific departments(Tian, Ma, Liang, Kwok, &Liu, plicants and affiliations of proposals assigned to reviewers. The 2005). The general office and bure mainly in charge of pol limitations of this strategy call for another approach in this con- icy making, operational management, administrative work and re- text. In this research, a new assignment strategy is proposed which lated affaires. The scientific departments are the key parts of NSFC. is based on grouping proposals first and then search qualified and they are responsible for the selection and management of re- reviewers for each proposal group(shown in Fig. 1B). To the best search projects. The scientific departments include Department of of our knowledge, it has never been addressed in the literature. Mathematicaland Physical Sciences, Department of Chemical Sciences, Furthermore, previous research usually used exact algorithms Department of Life Sciences, Department of Earth Sciences, Depart to solve the reviewer assignment problem. But it may be difficult ment of Engineering and Materials Sciences, Department of Informa- to solve the assignment problem using exact algorithms when tion Sciences and Department of Management Sciences. These seven the numbers of both proposals and reviewers are very large. Genet- scientific departments are further divided into divisions focusing various assignment and combinational applications where it dem- Management Sciences is further divided into three divisions: Man- onstrated satisfactory performances(Deep Das, 2008: Harper, de agement Science and Engineering, Macro Management and Policy Senna, vieira, Shahani, 2005: Huang Lim, 2006). While genetic and Business Administration algorithms in their elementary forms can be designed to tackle a here are various categories of programs in NSFC. The general real-world problem, the incorporation of domain knowledge and Program(including Project for young scientists'fund and Project local search techniques may improve the computational perfor- for developing regions)is the major one. The number of proposals mance significantly submitted to NSFC for the General Program increased dramati- This paper proposes an integrated approach assisting in assign- cally from 23, 636 in year 2001 to 73, 785 in year 2008(see ing proposals to reviewers where proposals need to be partitioned Fig. 2). Note that the average funded rate( funded over submit- into groups. The proposed approach facilitates the reviewer assign- ted )in 2005-2008 is only about 18%. In order to support and fi- nent through the following steps: identify valid proposals and nance the most promising proposals a limited budget, a viewers, classify proposals and reviewers according to their dis- fair and unbiased project selection pi ciplines, partition proposals into groups and assign reviewers to one of the most important tasks is to approprlate review proposal groups. Knowledge rules and models are used to support ers to proposals Proposals Reviewers Proposals Grouping Reviewers (A)Individual proposal assignment (B)Proposal assignment based on grouping them first

assignment of candidate reviewers for each proposal, using either one or both of these approaches (Papagelis, Plexousakis, & Niko￾laou, 2005; Sun et al., 2008). According to our literature review, the majority of existing ap￾proaches either from information retrieval stream or from optimi￾zation stream are based on the same strategy which seeks to search appropriate reviewers for each individual proposal (as shown in Part A of Fig. 1). This strategy is in an attempt to ensure that every reviewer has sufficient expertise to evaluate the merits of the proposals assigned to him/her. This strategy has several lim￾itations: first, selecting appropriate reviewers for every individual proposal is time-consuming. For large funding agencies, the num￾ber of proposals received and the number of qualified reviewers can be very large. Besides, the review process must be usually com￾pleted with a tight deadline (Wang et al., 2008). Thus, partitioning proposals into groups and searching reviewers for each proposal group can be an efficient alternative. The time spent on reviewer assignment can be reduced by assigning reviewers to proposal groups instead of individual proposals. Second, assigning individ￾ual proposals to reviewers can hardly meet the specific require￾ments of the funding agencies, e.g. balancing the backgrounds of applicants and affiliations of proposals assigned to reviewers. The limitations of this strategy call for another approach in this con￾text. In this research, a new assignment strategy is proposed which is based on grouping proposals first and then search qualified reviewers for each proposal group (shown in Fig. 1B). To the best of our knowledge, it has never been addressed in the literature. Furthermore, previous research usually used exact algorithms to solve the reviewer assignment problem. But it may be difficult to solve the assignment problem using exact algorithms when the numbers of both proposals and reviewers are very large. Genet￾ic algorithm (GA) has been widely used as a search algorithm in various assignment and combinational applications where it dem￾onstrated satisfactory performances (Deep & Das, 2008; Harper, de Senna, Vieira, & Shahani, 2005; Huang & Lim, 2006). While genetic algorithms in their elementary forms can be designed to tackle a real-world problem, the incorporation of domain knowledge and local search techniques may improve the computational perfor￾mance significantly. This paper proposes an integrated approach assisting in assign￾ing proposals to reviewers where proposals need to be partitioned into groups. The proposed approach facilitates the reviewer assign￾ment through the following steps: identify valid proposals and reviewers, classify proposals and reviewers according to their dis￾ciplines, partition proposals into groups and assign reviewers to proposal groups. Knowledge rules and models are used to support the aforementioned decision process, including the matching de￾gree calculation model, the proposal grouping model and the assignment model to assign reviewers to proposal groups, etc. The remainder of this paper is organized as follows. The re￾search background is introduced in Section 2. Section 3 proposes an approach for solving the assignment problem. Section 4 pre￾sents a prototype system based on the proposed approach. Section 5 validates the proposed approach and discusses the potential application in government funding agencies. Section 6 concludes the paper. 2. Background 2.1. Background of NSFC As one of the largest and most reputable research funding agen￾cies in China, NSFC (National Natural Science Foundation of China) aims to fund research projects that have great potential of scien￾tific and social impacts. NSFC has one general office, five bureaus, and seven scientific departments (Tian, Ma, Liang, Kwok, & Liu, 2005). The general office and bureaus are mainly in charge of pol￾icy making, operational management, administrative work and re￾lated affaires. The scientific departments are the key parts of NSFC, and they are responsible for the selection and management of re￾search projects. The scientific departments include Department of Mathematical and Physical Sciences, Department of Chemical Sciences, Department of Life Sciences, Department of Earth Sciences, Depart￾ment of Engineering and Materials Sciences, Department of Informa￾tion Sciences and Department of Management Sciences. These seven scientific departments are further divided into divisions focusing on more specific research areas. For example, the Department of Management Sciences is further divided into three divisions: Man￾agement Science and Engineering, Macro Management and Policy and Business Administration. There are various categories of programs in NSFC. The General Program (including Project for young scientists’ fund and Project for developing regions) is the major one. The number of proposals submitted to NSFC for the General Program increased dramati￾cally from 23,636 in year 2001 to 73,785 in year 2008 (see Fig. 2). Note that the average funded rate (funded over submit￾ted) in 2005–2008 is only about 18%. In order to support and fi- nance the most promising proposals within a limited budget, a fair and unbiased project selection process is necessary where one of the most important tasks is to assign appropriate review￾ers to proposals. (A) Individual proposal assignment (B) Proposal assignment based on grouping them first Proposals Reviewers Proposals’ Grouping Reviewers Fig. 1. Two strategies for assigning proposals. Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6949

Y. Xu et aL/ Expert Systems with Applications 37(2010)6948-6956 80000 O Submitted propos 73785 9329 39665 20000 2003 20042005200620072008 Year Fig.2.thEnumberofsubmittedandfundedproposalsfrom2001to2008.Sourcehttp://www.nsfc.gov.cn 2.2. Proposal assignment at NSFC In most situations, reviewing proposal is done on a voluntary basis. If the workload for a reviewer is too heavy. reviewers may refuse The proposal assignment at NSFC underwent two phases and the work assigned to them or pass on the job to their colleagues employed different strategies in the past as shown in Fig. 1. Ini- or subordinates. (4)For a fair review process, the number of tially, it adopted the strategy to find appropriate reviewers for each reviewers assigned to each proposal group should be about the dividual proposal. Subsequently, with large number of proposals same. For example, General Program in NSFC requires that each submitted and with special requirements of NSFC (e.g. balancing proposal be reviewed by five reviewers. (5) Reviewers who have the background information of applicants, balancing the affilia- conflicts of interests with an applicant in a proposal group should tions of proposals etc. ) it was necessary to group proposals first not be assigned to that group, e.g., the reviewers who co-authored and then search qualified reviewers for each proposal group a paper before with an applicant should be excluded f the review p ess, several issues should be considered when assigning reviewers 3. The proposed approach to proposal groups in NSFC:(1)The composition of proposals in The basic idea of our proposed approach is p proposals ants of proposals in each proposal group are from different uni- first and then assign reviewers to each proposal t versities or institutions. (2)The number of proposals in each proposed approach is composed of six steps as shown in Fig. 3. A group should be about the same. otherwise, there are proposal detailed explanation of each step is provided in the following groups with large group size and small group size. Under this con- suDs text, it is possible that some reviewers receive a large number of proposals, even when the number of proposal groups assigned to 3. 1. Step 1. Identifying valid proposals and reviewe each reviewer is same. (3)To ensure that reviewers have sufficient time to complete their review task, the number of proposal groups The aim of this step is to identify valid proposals and reviewers assigned to each reviewer should not exceed a certain number In NSFC, there are various types of programs such as General Pro (e.g. 16 proposals in 2 proposal groups each with 8 proposals ). gram, Distinguished Young Scholar Program, etc. The requirements validity identification Classification based on Grouping proposals up proposals based on ify valid proposals an fy proposals and reviewers research disciplines Result adjustment Reviewer assignment calculation Adjust assignment results by Assign reviewers to proposal between proposal groups and groups Fig 3. The assignment process

2.2. Proposal assignment at NSFC The proposal assignment at NSFC underwent two phases and employed different strategies in the past as shown in Fig. 1. Ini￾tially, it adopted the strategy to find appropriate reviewers for each individual proposal. Subsequently, with large number of proposals submitted and with special requirements of NSFC (e.g., balancing the background information of applicants, balancing the affilia￾tions of proposals etc.), it was necessary to group proposals first and then search qualified reviewers for each proposal group. The assignment results reflect the fairness of the review process to some extent. In order to ensure a fair and unbiased review pro￾cess, several issues should be considered when assigning reviewers to proposal groups in NSFC: (1) The composition of proposals in each group need to be as diverse as possible. For example, appli￾cants of proposals in each proposal group are from different uni￾versities or institutions. (2) The number of proposals in each group should be about the same. Otherwise, there are proposal groups with large group size and small group size. Under this con￾text, it is possible that some reviewers receive a large number of proposals, even when the number of proposal groups assigned to each reviewer is same. (3) To ensure that reviewers have sufficient time to complete their review task, the number of proposal groups assigned to each reviewer should not exceed a certain number (e.g., 16 proposals in 2 proposal groups each with 8 proposals). In most situations, reviewing proposal is done on a voluntary basis. If the workload for a reviewer is too heavy, reviewers may refuse the work assigned to them or pass on the job to their colleagues or subordinates. (4) For a fair review process, the number of reviewers assigned to each proposal group should be about the same. For example, General Program in NSFC requires that each proposal be reviewed by five reviewers. (5) Reviewers who have conflicts of interests with an applicant in a proposal group should not be assigned to that group, e.g., the reviewers who co-authored a paper before with an applicant should be excluded. 3. The proposed approach The basic idea of our proposed approach is to group proposals first and then assign reviewers to each proposal group. The entire proposed approach is composed of six steps as shown in Fig. 3. A detailed explanation of each step is provided in the following subsections. 3.1. Step 1. Identifying valid proposals and reviewers The aim of this step is to identify valid proposals and reviewers. In NSFC, there are various types of programs such as General Pro￾gram, Distinguished Young Scholar Program, etc. The requirements 27590 31791 39665 49329 58811 4435 5808 6359 7711 9111 10271 11608 14355 64649 23636 73785 Year Number Submitted proposals Funded projects Fig. 2. The number of submitted and funded proposals from 2001 to 2008. Source: http://www.nsfc.gov.cn Validity identification Identify valid proposals and reviewers Grouping proposals Group proposals based on criteria predetermined by managers Matching degree calculation Calculate matching degree between proposal groups and reviewers Result adjustment Adjust assignment results by managers Reviewer assignment Assign reviewers to proposal groups Classification based on disciplines Classify proposals and reviewers according to their research disciplines Fig. 3. The assignment process. 6950 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 Table Sample rules for identifying valid proposals. Sample rules for classifying reviewers. Rules for general progra Suppose that there are k discipline areas, and Dx g denotes reviewer j (k=1,2,- K): Assume that there are reviewers THEN C=1, 2,-J): g denotes the mth research area of reviewer R] S denotes the IF Unregistered(PD)or Unregistered (affiliated organization eviewer sets of discipline area k, then Sk can be calculated as follows: THEN Status= incomplete or k=1 to K IF Position of Pl= Full-Time Graduate student Forj=1 to J OR Position of Pl= Part-Time Graduate Student If R #(Categorized reviewers] Then umber of discipline areas of reviewer Ry IF PI in Bad Reputation List Record If R f= D Then THEN Status undetermined ill be added to Sk End If Next for applicants vary for different programs. For example, in the dis en tinguished Young Scholar Program, the age of the applicants should R, will be added to set(Categorized reviewers] be less than 35 years old. After proposals are submitted to NSFC, End If knowledge rules(see Table 1 for an example)can help managers End if plete, undetermined and valid. If a proposal is valid. it can be di- Next incomplete and undetermined proposals are rejected or returned to applicants for further considerations, and for possible resubmis- weights are determined in advance. The detail of the model and al,2002) the solution are presented in our previous research(Fan, Chen, iewers are selected based on their recent research publica Ma, Zhu, 2008). The goal of this step is to guarantee that the com- in research, e.g. more published papers and more funded projects, of NFc f the proposal groups is consistent with the requirements tions and funded projects. Experts who have better track records position have higher probability to be selected as reviewers, while those who are not active in the past few years are usually excluded 3. 4. Step 4. Calculating the matching degree (Sun, Ma, Fan, Wang. 2008). Knowledge rules can be used to facilitate identifying valid reviewers(see Table 2 for an example). One of the most important issues in reviewer assignment proce dure is to determine how good a reviewer matches a proposal 3. 2. Step 2. Classifying proposals and reviewers in terms of their group in terms of the extent to which the reviewer is familiar with he research area of the corresponding proposals. This judgment can be extracted ad hoc via expert knowledge, but it is impractical After identifying valid reviewers and proposals, the next step is to do so if there are a large number of proposal group and reviewer lassify these proposals and reviewers according to their disc pline areas. As mentioned before, there are seven scientific depart reviewers can be calculated as follows ents in NSFC. The departments are composed of different divisions. The divisions are further divided into many disciplines 3. 4.1. Calculating the matching degree between proposal groups and (Sun et al., 2008). NSFC maintains a dictionary of discipline areas reviewers which is arranged in a tree structure. This discipline dictionary is To determine the expertise level of reviewers in NSFC, review released to applicants and reviewers. It facilitates applicants to de- ers are required to fill in a form declaring their expertise level in clare more carefully the discipline areas that their proposals belong the discipline areas with supporting evidence of publication re- to The reviewers also declare their research areas according to the cords. The expertise level of the reviewers is measured along a same dictionary. However, due to large number of proposals, scale using 1-3, where level 3 signifies that a reviewer is very reviewers and discipline areas, it is a time-consuming process for familiar with the corresponding research area, level 2 familiar the managers to manually create the appropriate matching pro- and level 1 less familiar. The matching degree between a specific osal sets and reviewer sets in each discipline area. The knowledge reviewer and a proposal group can be calculated as follows ules can thus be used to aid the managers to classify proposal If we use Ck to denote the expertise level of reviewer j to disci- and reviewers(see Table 3 as an example). 3.3. Step 3. Partitioning proposals into proposal groups 3, if reviewer is"very familiar"with discipline area k Ck= 2, if reviewer is"familiar"with discipline area k As mentioned in Section 2, it is necessary to consider the fund 1. if reviewer is"less familiar"with discipline area k ing policies and requirements of NSFC in the initial grouping pro cess before assigning proposals to reviewers. The attributes that dgk is a binary value whose value is 1 if proposal g belongs to disci- are considered when grouping proposals and their corresponding pline k, otherwise, it is zero i proposal group(i=1,2,., n). The matching degree between proposal group i and reviewerjis Table 2 Sample rules for identifying valid reviewers. obtained by Mingei Max( Ck*dgk). The matching degree between an individual proposal g(in group i) and reviewer j is measured by ot activ Max(Ci*dgk). It uses one of a specific proposal's discipline with which a reviewer is most familiar to obtain the matching degree ad Reputation Record us invalid between a proposal and a reviewer while the minimal value of matching degree between a reviewer and proposals in a particular

for applicants vary for different programs. For example, in the Dis￾tinguished Young Scholar Program, the age of the applicants should be less than 35 years old. After proposals are submitted to NSFC, knowledge rules (see Table 1 for an example) can help managers to classify the proposals into four categories, i.e., invalid, incom￾plete, undetermined and valid. If a proposal is valid, it can be di￾rectly passed to the next selection phase. While invalid, incomplete and undetermined proposals are rejected or returned to applicants for further considerations, and for possible resubmis￾sion (Tian et al., 2002). Reviewers are selected based on their recent research publica￾tions and funded projects. Experts who have better track records in research, e.g. more published papers and more funded projects, have higher probability to be selected as reviewers, while those who are not active in the past few years are usually excluded (Sun, Ma, Fan, & Wang, 2008). Knowledge rules can be used to facilitate identifying valid reviewers (see Table 2 for an example). 3.2. Step 2. Classifying proposals and reviewers in terms of their discipline areas After identifying valid reviewers and proposals, the next step is to classify these proposals and reviewers according to their disci￾pline areas. As mentioned before, there are seven scientific depart￾ments in NSFC. The departments are composed of different divisions. The divisions are further divided into many disciplines (Sun et al., 2008). NSFC maintains a dictionary of discipline areas which is arranged in a tree structure. This discipline dictionary is released to applicants and reviewers. It facilitates applicants to de￾clare more carefully the discipline areas that their proposals belong to. The reviewers also declare their research areas according to the same dictionary. However, due to large number of proposals, reviewers and discipline areas, it is a time-consuming process for the managers to manually create the appropriate matching pro￾posal sets and reviewer sets in each discipline area. The knowledge rules can thus be used to aid the managers to classify proposals and reviewers (see Table 3 as an example). 3.3. Step 3. Partitioning proposals into proposal groups As mentioned in Section 2, it is necessary to consider the fund￾ing policies and requirements of NSFC in the initial grouping pro￾cess before assigning proposals to reviewers. The attributes that are considered when grouping proposals and their corresponding weights are determined in advance. The detail of the model and the solution are presented in our previous research (Fan, Chen, Ma, & Zhu, 2008). The goal of this step is to guarantee that the com￾position of the proposal groups is consistent with the requirements of NSFC. 3.4. Step 4. Calculating the matching degree One of the most important issues in reviewer assignment proce￾dure is to determine how good a reviewer matches a proposal group in terms of the extent to which the reviewer is familiar with the research area of the corresponding proposals. This judgment can be extracted ad hoc via expert knowledge, but it is impractical to do so if there are a large number of proposal group and reviewer combinations. The matching degree between proposal groups and reviewers can be calculated as follows. 3.4.1. Calculating the matching degree between proposal groups and reviewers To determine the expertise level of reviewers in NSFC, review￾ers are required to fill in a form declaring their expertise level in the discipline areas with supporting evidence of publication re￾cords. The expertise level of the reviewers is measured along a scale using 1–3, where level 3 signifies that a reviewer is very familiar with the corresponding research area, level 2 familiar and level 1 less familiar. The matching degree between a specific reviewer and a proposal group can be calculated as follows. If we use Cjk to denote the expertise level of reviewer j to disci￾pline area k, then: Cjk ¼ 3; if reviewer is “very familiar” with discipline area k 2; if reviewer is “familiar” with discipline area k 1; if reviewer is “less familiar” with discipline area k 8 >< >: dgk is a binary value whose value is 1 if proposal g belongs to disci￾pline k, otherwise, it is zero. i proposal group (i = 1,2,...,n). The matching degree between proposal group i and reviewer j is obtained by Ming2iMaxk(Cjk dgk). The matching degree between an individual proposal g (in group i) and reviewer j is measured by Max k ðCjk dgkÞ. It uses one of a specific proposal’s discipline with which a reviewer is most familiar to obtain the matching degree between a proposal and a reviewer. While the minimal value of matching degree between a reviewer and proposals in a particular Table 1 Sample rules for identifying valid proposals. Rules for general program IF Number of on-going projects + Number of Submitted proposals > 2 THEN Status = Undetermined IF Unregistered (PI) or Unregistered (affiliated organization) THEN Status = incomplete IF Position of PI = Full-Time Graduate Student OR Position of PI = Part-Time Graduate Student OR Position of PI = Retired THEN Status = Invalid IF PI in Bad Reputation List Record THEN Status = undetermined Table 2 Sample rules for identifying valid reviewers. IF reviewer not active in recent five years THEN Status = undetermined IF reviewer in Bad Reputation Record THEN Status = invalid Table 3 Sample rules for classifying reviewers. Suppose that there are K discipline areas, and Dk denotes discipline area k (k = 1,2,...,K); Assume that there are J reviewers; Rj denotes reviewer j (j = 1,2,...,J); Rn j denotes the nth research area of reviewer Rj, Sk denotes the reviewer sets of discipline area k, then Sk can be calculated as follows: For k = 1 to K For j = 1 to J If Rj R {Categorized reviewers} Then N = Number of discipline areas of reviewer Rj For n = 1 to N If Rn j ¼ Dk Then Rj will be added to Sk End Loop End If Next If n = N Then Rj will be added to set {Categorized reviewers} End If End If Next Next Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6951

Y. Xu et aL/ Expert Systems with Applications 37(2010)6948-6956 group is used to measure the matching degree between a proposal Initialize population P(r) based on GRASP group and a reviewer. 3. 4.2. Eliminating possible conflicts of interests Select parents from P(1) In order to ensure the fairness of the project selection, potential conflicts of interests between reviewers and applicants must be avoided as much as possible. For example, the applicants and on operator on parents eviewers cannot come form same institution or have co-authe relationship. If conflicts of interests between any applicant of proposal group and a reviewer exist, then the matching degree be- ween this reviewer and the proposal group is set to be o or a neg- Applyrepair operator on infeasible children tive number so as to avoid the matching. 3.5. Step 5. Assigning reviewers to proposal groups Use local search to improve the quality of children In step 5, the proposal groups generated in step 3 are assigned Evaluate the fitness of childre to reviewers based on the matching degree between proposal whether groups and reviewers calculated in step 4 and other considerations as mentioned in Section 2. Recall that the research problem is to find the most qualified reviewers for each proposal group. That t=t+I is, maximize the matching degree between the proposal groups d reviewers while satisfying the funding constraints. This is ex- pressed in an assignment model as discussed in Section 3.5.1. Then a hybrid approach based on gRASP (Greedy randomized Adaptive Search Procedure)and genetic algorithm is proposed to solve the assignment model for the large volume problems in the following 3.5.1. The assignment model Let n be the number of proposal groups needed to be assigned m be the total number of available reviewers, i be the index of pro- posal groups (i=1,2,., n) be G=1, 2,..., m), a be the required number of reviewers for evaluat- Fig 4. The structure of the hy brid GRASP and gA ing each proposal group(in most funding it is usually 3 or 5) and b be the maximum number of proposal groups that a re- ewer can be assigned to. xy is a binary variable whose value is g algorithm may not be superior to other heuristic 1 if group i is assigned to reviewer j, and 0 otherwise. The mathe algorit when a genetic algorithm is combined with a local matical model can be formulated as follows: search he efficiency of the algorithm can be improve Ahuja, Orlin, Tiwari, 2000). A combination of hybrid GRASI Maximize∑∑Mn2sMax(C*4)k and genetic algorithm can be used to solve the above mathematical model while overcoming the weaknesses of the general GA. It Subject to a vi (1) cludes the steps of initialization, selection, crossover, mutation and other operators as shown in Fig 4 ∑刈≤b可 3.5.3. The hybrid GRASP and GA Xi=0.1 (3) 3.5.3.1. Representation. A matrix whose elements are either 0 1 can be used to represent a solution to the reviewer assignmen Constraint(1)guarantees that the number of proposal groups as- to the jth reviewer, then the value of the corresponding element of signed to each reviewer should not exceed a predetermined number to ensure the review quality when the number of proposals and the solution matrix is 1 or o otherwise. The combination of 0 and 1 reviewers is small, the exact algorithm can work ell to solvi variables in the matrix do are as- signed to which reviewers(as shown in Fig. 5). When the matrix ciently by exact algorithm in reasonable time due to the dual prob- is too sparse, another representation technique is used where the lems of large volume of proposals and reviewers and limited available time. Thus, we propose here a heuristic algorithm based on a hybrid GRASP and genetic algorithm for an efficient solution R1R2R3….R。 when the number of proposals and reviewers is large G1101 G,01 3.5.2. The proposed solution to the assignment model G3110 Genetic algorithms have been applied to several kinds of com- tory performance(Huang Lim, 2006: Nebro, Luque, Luna, Alba 2008: Tang, Quek, Ng, 2005). It has also been shown that a Fig. 5. Representation of an individuals chromosome

group is used to measure the matching degree between a proposal group and a reviewer. 3.4.2. Eliminating possible conflicts of interests In order to ensure the fairness of the project selection, potential conflicts of interests between reviewers and applicants must be avoided as much as possible. For example, the applicants and reviewers cannot come form same institution or have co-author relationship. If conflicts of interests between any applicant of a proposal group and a reviewer exist, then the matching degree be￾tween this reviewer and the proposal group is set to be 0 or a neg￾ative number so as to avoid the matching. 3.5. Step 5. Assigning reviewers to proposal groups In step 5, the proposal groups generated in step 3 are assigned to reviewers based on the matching degree between proposal groups and reviewers calculated in step 4 and other considerations as mentioned in Section 2. Recall that the research problem is to find the most qualified reviewers for each proposal group. That is, maximize the matching degree between the proposal groups and reviewers while satisfying the funding constraints. This is ex￾pressed in an assignment model as discussed in Section 3.5.1. Then a hybrid approach based on GRASP (Greedy Randomized Adaptive Search Procedure) and genetic algorithm is proposed to solve the assignment model for the large volume problems in the following subsections. 3.5.1. The assignment model Let n be the number of proposal groups needed to be assigned, m be the total number of available reviewers, i be the index of pro￾posal groups (i = 1,2,...,n),j be the index of reviewers (j = 1,2,...,m), a be the required number of reviewers for evaluat￾ing each proposal group (in most funding agencies, it is usually 3 or 5), and b be the maximum number of proposal groups that a re￾viewer can be assigned to. xij is a binary variable whose value is 1 if group i is assigned to reviewer j, and 0 otherwise. The mathe￾matical model can be formulated as follows: Maximize X i2I X j2J ½Ming2iMaxkðCjk dgkÞxij Subject to X j xij ¼ a 8i ð1Þ X i xij 6 b 8j ð2Þ xij ¼ 0; 1 ð3Þ Constraint (1) expresses the requirement that each group of propos￾als should be evaluated by a reviewers. Constraint (2) implies that each reviewer should be assigned at most to b proposal groups. Constraint (1) guarantees that the number of proposal groups as￾signed to each reviewer should not exceed a predetermined number to ensure the review quality. When the number of proposals and reviewers is small, the exact algorithm can work very well to solve the above model. However, the above model cannot be solved effi- ciently by exact algorithm in reasonable time due to the dual prob￾lems of large volume of proposals and reviewers and limited available time. Thus, we propose here a heuristic algorithm based on a hybrid GRASP and genetic algorithm for an efficient solution when the number of proposals and reviewers is large. 3.5.2. The proposed solution to the assignment model Genetic algorithms have been applied to several kinds of com￾binatorial optimization problems and have demonstrated satisfac￾tory performance (Huang & Lim, 2006; Nebro, Luque, Luna, & Alba, 2008; Tang, Quek, & Ng, 2005). It has also been shown that a general genetic algorithm may not be superior to other heuristic algorithms. But when a genetic algorithm is combined with a local search strategy, the efficiency of the algorithm can be improved (Ahuja, Orlin, & Tiwari, 2000). A combination of hybrid GRASP and genetic algorithm can be used to solve the above mathematical model while overcoming the weaknesses of the general GA. It in￾cludes the steps of initialization, selection, crossover, mutation and other operators as shown in Fig. 4. 3.5.3. The hybrid GRASP and GA 3.5.3.1. Representation. A matrix whose elements are either 0 or 1 can be used to represent a solution to the reviewer assignment problem, which is referred to as chromosome in genetic algorithm. Using R1, R2,...,Rm, and G1, G2,...,Gn to represent reviewers and proposal groups, respectively, if the ith proposal group is assigned to the jth reviewer, then the value of the corresponding element of the solution matrix is 1 or 0 otherwise. The combination of 0 and 1 variables in the matrix determines which proposal groups are as￾signed to which reviewers (as shown in Fig. 5). When the matrix is too sparse, another representation technique is used where the Initialize population P(t) based on GRASP Select parents from P(t) Apply crossover and mutation operator on parents to yield children Apply repair operator on infeasible children Use local search to improve the quality of children Evaluate the fitness of children to determine whether replace the individuals or not t=t+1 Stop criterion satified Stop Yes No Fig. 4. The structure of the hybrid GRASP and GA. 123 1 2 3 ... 1 0 1 ... 0 0 1 1 ... 0 1 1 0 ... 0 ... ... ... ... ... ... 0 0 1 ... 1 m n RRR R G G G G Fig. 5. Representation of an individual’s chromosome. 6952 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956