《电子商务 E-business》阅读文献：A Survey on Reviewer Assignment Problem

A Survey on Reviewer Assignment Problem Fan wang. Ben Chen. and Zhaowei Miao I School of Business, Sun Yat-Sen University, Guangzhou, PRChina chenben819@gmail co M School, Xiamen University, Xiamen, P R China Abstract. Research into Reviewer Assignment Problem(RAP)is still in its early stage but there is great world-wide interest, as the foregoing process of peer-review which is the brickwork of science authentication. The RAP ap- proach can be divided into three phases: identifying assignment procedure omputing the matching degree between manuscripts and reviewers, and opti- mizing the assignment so as to achieve the given objectives. Methodologies for addressing the above three phases have been developed from a variety of re- search disciplines, including information retrieval, artificial intelligent, opera- tions research, etc. This survey is not only to cover variations of rAP that have ppeared in the literature, but also to identify the practical challenge and current progress for developing intelligent RAP systems Keywords: reviewer assign information retrieval; conference system. 1 Introduction Assigning submitted manuscripts to reviewers known as Reviewer Assignment Prob lem(RAP), is an important but tough task for journal editors, conference program chairs, and research councils. The essence of RAP can be divided into three phases: (1)identifying the assignment procedure, (2)computing the match between manu- scripts and reviewers, and (3)optimizing the assignment so as to maximize the match within the feasible restriction Reviewer assignment, traditionally handled by a single person (or at most a few people), is approached to satisfy four conditions: matching manuscripts and review- ers, fulfilling manuscript slots, balancing reviewers' workload and avoiding conflicts of interest. As reviewer assignment must be completed under severe timing con- straints, along with a very large number of submissions arriving near an announced deadline, it makes RAP a stressed dual problem of time and labor intensive. In re- sponse to the need of automatic mechanism, a number of studies have addressed the assignment solutions since Dumais and Nielsens[14] breakthrough orresponding Author. This research is supported by National Natural Science Foundation of China under Project No. 60704048. The author Dr. Zhaowei Miao thanks the support by Hu manities and Social Science Project(No. 07JC630047)of National Ministry of Education, China N.T. Nguyen et al.(Eds IEA/AIE 2008, LNAl 5027, PP. 718-727, 2008. o Springer-Verlag Berlin Heidelberg 2008

N.T. Nguyen et al. (Eds.): IEA/AIE 2008, LNAI 5027, pp. 718–727, 2008. © Springer-Verlag Berlin Heidelberg 2008 A Survey on Reviewer Assignment Problem Fan Wang1 , Ben Chen1,*, and Zhaowei Miao2 1 School of Business, Sun Yat-Sen University, Guangzhou, P.R. China chenben819@gmail.com 2 Management School, Xiamen University, Xiamen, P.R. China Abstract. Research into Reviewer Assignment Problem (RAP) is still in its early stage but there is great world-wide interest, as the foregoing process of peer-review which is the brickwork of science authentication. The RAP ap￾proach can be divided into three phases: identifying assignment procedure, computing the matching degree between manuscripts and reviewers, and opti￾mizing the assignment so as to achieve the given objectives. Methodologies for addressing the above three phases have been developed from a variety of re￾search disciplines, including information retrieval, artificial intelligent, opera￾tions research, etc. This survey is not only to cover variations of RAP that have appeared in the literature, but also to identify the practical challenge and current progress for developing intelligent RAP systems. Keywords: reviewer assignment; information retrieval; conference system. 1 Introduction Assigning submitted manuscripts to reviewers known as Reviewer Assignment Prob￾lem (RAP), is an important but tough task for journal editors, conference program chairs, and research councils. The essence of RAP can be divided into three phases: (1) identifying the assignment procedure, (2) computing the match between manu￾scripts and reviewers, and (3) optimizing the assignment so as to maximize the match within the feasible restriction. Reviewer assignment, traditionally handled by a single person (or at most a few people), is approached to satisfy four conditions: matching manuscripts and review￾ers, fulfilling manuscript slots, balancing reviewers’ workload and avoiding conflicts of interest. As reviewer assignment must be completed under severe timing con￾straints, along with a very large number of submissions arriving near an announced deadline, it makes RAP a stressed dual problem of time and labor intensive. In re￾sponse to the need of automatic mechanism, a number of studies have addressed the assignment solutions since Dumais and Nielsen’s [14] breakthrough. * Corresponding Author. This research is supported by National Natural Science Foundation of China under Project No. 60704048. The author Dr. Zhaowei Miao thanks the support by Hu￾manities and Social Science Project (No. 07JC630047) of National Ministry of Education, China

A Survey on Reviewer Assignment Problem 719 In general, RAP described in this paper, should be relevant whenever one has two large sets of objects that need to be matched such that each object in one set gets as- mall number of objects from the other, and have many practical appl tions, such as resource allocation(matching funding agencies to research projects[10] scheduling on parallel machines [13], assigning managers to construction projects [27], classroom assignment [8]), staff scheduling (assigning graduating students to interviewer, assigning press releases to newspaper reporters, matching staff to pro- jects in consulting companies [7], crew scheduling in airline companies [19], posting military servicemen [3, 25)), and construction decision making(corrective action rec ommendation for material management (38)) This wide range of real-world applications has constituted a major motivation for scholars in developing solutions for RAP. Reviewer assignment has been discussed as a process of peer-review in the voice of improving review mechanism. One of the earliest papers that addressed an assignment solution is by Dumais and Nielsen [14] who presented the matching degree computation by Latent Semantic Indexing(LSD), which was refined by Yarowsky and Florian [41] on using reviewers' online publica tion instead of their autobiography. Different from content-based Information Re trieval (IR), Watanabe et al. [39 raised the matching degree computation by constructing collaborative network, which leads to the hybrid approach of the two technologies. On the other hand, Operations Research(OR) has been used to build the assignment models and design algorithms of optimization RAP, which has been widely approached along different disciplines, can be applied into various application fields and represented by multifarious items, thus making it more difficult to search for previous work on the topic. This paper is intended to pro- vide a comprehensive survey according to the researches which have appeared in the literature sorting by the three phases of RaP. The value of such a study is in provid- ing an opportunity to reflect on what has been achieved, to identify gaps that need to be addressed. and to set direction for future research The reminder of this paper is organized as follows. Section 2 provides a review on the assignment procedure. Section 3 presents IR technology for computing matching degree between manuscripts and reviewers. Section 4 is a summary on the method- ologies for optimization modeling. Finally, conclusions are presented and suggestions are made for further research 2 Reviewer Assignment Procedure Researches defended on reviewer assignment procedure involve the specification of identification, and its influence in the peer-review mode( utions of qualified reviewer four issues: assignment criteria, assignment processes, so The assignment criteria play an important role in the methodology and procedure of reviewer assignment, and indirectly affect the peer-review quality. Fixed assign ment criteria are widely used in the science community, which require authors to select keywords from one list of disciplines or prorate the keywords. Fixed lists make up of two or three columns sometimes(e.g." Methodology","Application"and"Oth ers"[33D, which makes the chosen keywords more comprehensively. As it is argued that the update of fixed lists cannot catch up with the development of disciplines, the

A Survey on Reviewer Assignment Problem 719 In general, RAP described in this paper, should be relevant whenever one has two large sets of objects that need to be matched such that each object in one set gets as￾signed a small number of objects from the other, and have many practical applica￾tions, such as resource allocation (matching funding agencies to research projects[10], scheduling on parallel machines [13], assigning managers to construction projects [27], classroom assignment [8]), staff scheduling (assigning graduating students to interviewer, assigning press releases to newspaper reporters, matching staff to pro￾jects in consulting companies [7], crew scheduling in airline companies [19], posting military servicemen [3,25]), and construction decision making (corrective action rec￾ommendation for material management [38]). This wide range of real-world applications has constituted a major motivation for scholars in developing solutions for RAP. Reviewer assignment has been discussed as a process of peer-review in the voice of improving review mechanism. One of the earliest papers that addressed an assignment solution is by Dumais and Nielsen [14], who presented the matching degree computation by Latent Semantic Indexing (LSI), which was refined by Yarowsky and Florian [41] on using reviewers’ online publica￾tion instead of their autobiography. Different from content-based Information Re￾trieval (IR), Watanabe et al. [39] raised the matching degree computation by constructing collaborative network, which leads to the hybrid approach of the two technologies. On the other hand, Operations Research (OR) has been used to build the assignment models and design algorithms of optimization. RAP, which has been widely approached along different disciplines, can be applied into various application fields and represented by multifarious items, thus making it more difficult to search for previous work on the topic. This paper is intended to pro￾vide a comprehensive survey according to the researches which have appeared in the literature sorting by the three phases of RAP. The value of such a study is in provid￾ing an opportunity to reflect on what has been achieved, to identify gaps that need to be addressed, and to set direction for future research. The reminder of this paper is organized as follows. Section 2 provides a review on the assignment procedure. Section 3 presents IR technology for computing matching degree between manuscripts and reviewers. Section 4 is a summary on the method￾ologies for optimization modeling. Finally, conclusions are presented and suggestions are made for further research. 2 Reviewer Assignment Procedure Researches defended on reviewer assignment procedure involve the specification of four issues: assignment criteria, assignment processes, solutions of qualified reviewer identification, and its influence in the peer-review model. The assignment criteria play an important role in the methodology and procedure of reviewer assignment, and indirectly affect the peer-review quality. Fixed assign￾ment criteria are widely used in the science community, which require authors to select keywords from one list of disciplines or prorate the keywords. Fixed lists make up of two or three columns sometimes (e.g. “Methodology”, “Application” and “Oth￾ers” [33]), which makes the chosen keywords more comprehensively. As it is argued that the update of fixed lists cannot catch up with the development of disciplines, the

F Wang B. Chen. and Z miao unfixed criteria are also used in some conference procedure, especially when the dis- cipline information is not integrated or hard to represent by several keywords. In that case, IR technology is applied to compute the similarity between manuscripts and reviewers' biographies [14]. In the mean while, data mining is used to extract the keyword-list by doing unsupervised clustering or supervised learning by using pervi ous accepted papers as training set [5] Reviewer assignment is the foregoing process of peer-review in the manuscr selection procedure, which directly influences the review result and evaluation aggre gating. Conference chair usually works on the reviewer assignment with the regis tered information of manuscripts and reviewers, and then solution is addressed within the certain community. Meanwhile, in the project selection of national funding com- mittee(e.g. NSF), the assignment is firstly run for an optimal solution, and new re- viewers are invited in case no satisfied solution exists. As the potential reviewer pool is large, committee chair never worries about reviewers workload intensive [21, 36 Moreover, some large science communities(e.g. AAAD)ask reviewers to bid on manuscripts by scanning abstracts, which is taken into consideration for the assign- ment. The bid behavior had been studied as human -factor noise which influences the preference very much rather than disciplines of manuscripts [3 Subjects of domain and conflicts of interest are the two main factors in identifying qualified reviewers. Collecting information of these two factors, known as building of knowledge-database, attracts attention of many conference chairs. Domain informa tion can be submitted by authors and reviewers during registration; but information of conflict, including collaborative relation, student-advisor-relationship, colleague rela- tion, is hard to collect. Geller [15] raised this problem to challenge the Al committee to call for an intelligent solution. Furthermore, Geller and Scherl [16] described how to search Internet to generate a potential-reviewers-list There exists a rich body of literature on peer-review that point out the inadequacies of the current systems. Weber[40] presented his manifestos for changing the journal review processes, since the assignment between manuscripts and reviewers works irrationally and inefficiently. Casati et al.[9] asked for more awareness on the open efficient review model and the reasonable assigning manuscripts to reviewers using information technology along with internet. Some scholars argued that the automatic reviewer assignment approaches bereaved their rights on classifying their own prob- lems which were treated as the scientists'most precious possession [34]. And it is said that the taxonomy of disciplines is not changed momentarily, which makes some interdisciplinary researches and frontier of science will never be recognized by the 3 Assignment Based on Information Retrieval IR used on reviewer assignment focuses on the second phase of RAP, which is com- puting the matching degree between manuscripts and reviewers. This phase had been approached mainly in four ways: content-based IR, collaborative filtering, hybrid approach of the former two and data mining One of the earliest RAP solutions found in literature is by IR, since inefficiency of ee scoring manually was firstly raised. Using the content-based IR

720 F. Wang, B. Chen, and Z. Miao unfixed criteria are also used in some conference procedure, especially when the dis￾cipline information is not integrated or hard to represent by several keywords. In that case, IR technology is applied to compute the similarity between manuscripts and reviewers’ biographies [14]. In the mean while, data mining is used to extract the keyword-list by doing unsupervised clustering or supervised learning by using pervi￾ous accepted papers as training set [5]. Reviewer assignment is the foregoing process of peer-review in the manuscripts selection procedure, which directly influences the review result and evaluation aggre￾gating. Conference chair usually works on the reviewer assignment with the regis￾tered information of manuscripts and reviewers, and then solution is addressed within the certain community. Meanwhile, in the project selection of national funding com￾mittee (e.g. NSF), the assignment is firstly run for an optimal solution, and new re￾viewers are invited in case no satisfied solution exists. As the potential reviewer pool is large, committee chair never worries about reviewers’ workload intensive [21,36]. Moreover, some large science communities (e.g. AAAI) ask reviewers to bid on manuscripts by scanning abstracts, which is taken into consideration for the assign￾ment. The bid behavior had been studied as human-factor noise which influences the preference very much rather than disciplines of manuscripts [31]. Subjects of domain and conflicts of interest are the two main factors in identifying qualified reviewers. Collecting information of these two factors, known as building of knowledge-database, attracts attention of many conference chairs. Domain informa￾tion can be submitted by authors and reviewers during registration; but information of conflict, including collaborative relation, student-advisor-relationship, colleague rela￾tion, is hard to collect. Geller [15] raised this problem to challenge the AI committee to call for an intelligent solution. Furthermore, Geller and Scherl [16] described how to search Internet to generate a potential-reviewers-list. There exists a rich body of literature on peer-review that point out the inadequacies of the current systems. Weber [40] presented his manifestos for changing the journal review processes, since the assignment between manuscripts and reviewers works irrationally and inefficiently. Casati et al. [9] asked for more awareness on the open efficient review model and the reasonable assigning manuscripts to reviewers using information technology along with internet. Some scholars argued that the automatic reviewer assignment approaches bereaved their rights on classifying their own prob￾lems which were treated as the scientists’ most precious possession [34]. And it is said that the taxonomy of disciplines is not changed momentarily, which makes some interdisciplinary researches and frontier of science will never be recognized by the corresponding committee. 3 Assignment Based on Information Retrieval IR used on reviewer assignment focuses on the second phase of RAP, which is com￾puting the matching degree between manuscripts and reviewers. This phase had been approached mainly in four ways: content-based IR, collaborative filtering, hybrid approach of the former two and data mining. One of the earliest RAP solutions found in literature is by IR, since inefficiency of matching degree scoring manually was firstly raised. Using the content-based IR

A Survey on Reviewer Assignment Problem 721 method known as LSI, Dumais and Nielsen [14] represented each manuscripts and reviewers'autobiography by a matrix containing nearly 100 item vectors of factors weight, the matching degree was computed as the dot product of the two matrixes Then, assignment was done by picking several reviewers from those with high match ing degree. A similar task was performed by Yarowsky and Florian [41], but review ers' biographies were replaced by their publications which were submitted by them- selves or downloaded from internet. Beginning with Dumais and Nielsens [14] paper, there are nine papers addressed the solution by using IR techniques. Table 1 reveals that content-based methodologies are acceptable, as text is the most important factor for manuscripts and reviewers' biographies, but not the only one for assignment Biswas and Hasan [5] compared the applicability of different content-based filterin and indicated that hybrid approaches might be a more comprehensive way Table 1. Review on the use of Ir solutions for rAp Methodology Yarowsky and Florian [41] Content-Based VSM/Naive Bayes Classifier Basu et al. [2] Popescul et al. [30] Hybrid Watanabe et al.[39 Collaborative Filtering Scale-free Network Hettich and Pazzani [21] Data Mining Data mining Rodriguez and Bollen [32] Collaborative Filtering Relative-rank Particle-Sw Algorithm Biswas and Hasan 5] Content-Based VSM( Comparison Study) Hvbrid Semantic web Studies by ir develop with the progresses of the technology itself. Content-based IR looks only at the contents of an artifact(e. g, the words on a paper), whereas col laborative filtering, which also consider the opinions of other like-minded people with respect to these artifacts, has been used to recommend NetNews articles [26], movies [1, 22], music[ll], and even jokes [18]. Scale-free network, which can continuously expand with the addition of new vertices, is a useful mechanism of collaborative fil tering. Watanabe et al. [39]had constructed a scale-free network whose vertices were keywords of reviewers'expertise and manuscripts' topic, and similarity between two keywords was the probability of connecting between the corresponding vertices. The matching degree is the weighted average of similarities between each pair of key words of manuscripts and reviewers. Instead of keywords, Rodriguez and Bollen [32] had approached a co-authorship network with vertices representing experts, edges representing a tie between two experts, and weights representing the strength of tie. with the application of collaborative filtering in other operations, a hybrid ap- roach was achieved before collaborative filtering by using co-authors and authors of reference to approach the collaborative method in the paper of Basu et al. [2]. Their framework provides a more flexible alternative to simple keyword-based search algo rithms and a less intrusive alternative to collaborative methods. Popescul et al. [30]

A Survey on Reviewer Assignment Problem 721 method known as LSI, Dumais and Nielsen [14] represented each manuscripts and reviewers’ autobiography by a matrix containing nearly 100 item vectors of factors weight, the matching degree was computed as the dot product of the two matrixes. Then, assignment was done by picking several reviewers from those with high match￾ing degree. A similar task was performed by Yarowsky and Florian [41], but review￾ers’ biographies were replaced by their publications which were submitted by them￾selves or downloaded from internet. Beginning with Dumais and Nielsen’s [14] paper, there are nine papers addressed the solution by using IR techniques. Table 1 reveals that content-based methodologies are acceptable, as text is the most important factor for manuscripts and reviewers’ biographies, but not the only one for assignment. Biswas and Hasan [5] compared the applicability of different content-based filtering and indicated that hybrid approaches might be a more comprehensive way. Table 1. Review on the use of IR solutions for RAP Study Methodology Techniques Dumais and Nielsen [14] Content-Based Latent Semantic Indexing Yarowsky and Florian [41] Content-Based VSM/Naive Bayes Classifier Basu et al. [2] Hybrid N/A Popescul et al. [30] Hybrid N/A Watanabe et al. [39] Collaborative Filtering Scale-free Network Hettich and Pazzani [21] Data Mining Data Mining Rodriguez and Bollen [32] Collaborative Filtering Relative-rank Particle-swarm Algorithm Biswas and Hasan [5] Content-Based VSM (Comparison Study) Cameron et al. [6] Hybrid Semantic Web Studies by IR develop with the progresses of the technology itself. Content-based IR looks only at the contents of an artifact (e.g., the words on a paper), whereas col￾laborative filtering, which also consider the opinions of other like-minded people with respect to these artifacts, has been used to recommend NetNews articles [26], movies [1,22], music[11], and even jokes [18]. Scale-free network, which can continuously expand with the addition of new vertices, is a useful mechanism of collaborative fil￾tering. Watanabe et al. [39] had constructed a scale-free network whose vertices were keywords of reviewers’ expertise and manuscripts’ topic, and similarity between two keywords was the probability of connecting between the corresponding vertices. The matching degree is the weighted average of similarities between each pair of key￾words of manuscripts and reviewers. Instead of keywords, Rodriguez and Bollen [32] had approached a co-authorship network with vertices representing experts, edges representing a tie between two experts, and weights representing the strength of tie. With the application of collaborative filtering in other operations, a hybrid ap￾proach was achieved before collaborative filtering by using co-authors and authors of reference to approach the collaborative method in the paper of Basu et al. [2]. Their framework provides a more flexible alternative to simple keyword-based search algo￾rithms and a less intrusive alternative to collaborative methods. Popescul et al. [30]

722 F Wang B Chen and Z miao proposed a unified probabilistic framework for combining content-based and collabo- rative IR by extending Hofmann's [23] aspect model to incorporate the information source among scholar, manuscript and manuscript content. Semantic Web technology was also brought to collect data, represent researches'expertise and co-authorship and find relevant reviewers in a peer-review setting [6]. As the increase of internet resource, data mining can be used to identify relative reviewers within an existing expert pool by mining online information through search engines, potential external reviewers can also be found in the online academic community In respond to the difficulty for a scientific community to agree up k eywords and maintain such a keyword database over time, Hettich and Pazzani [21] described the data mining method deployed at the U.S. NSF for assisting program directors in identifying reviewers for proposals. To the best of our knowledge, thei paper is the only one work found in literature which combined IR and optimization to achieve rap 4 Reviewer Assignment Optimization Optimization on reviewer assignment which mainly focus on the theory, modeling algorithm of assigning manuscripts to reviewers, can be viewed as an enhanced version of the Generalized Assignment Problem(GAP). However, most of these stud- ies are not based on the matching degree evaluated by IR, but simply in sense of simi- larity by the selected keywords. In making it easier to read, the notations used in the cited sources have, where necessary, been changed to try to keep consistent as those of traditional gap The RAP optimization we study is the following. We are given a set P=(l,,P) of manuscripts and a set R=(lr) of reviewers; and a parameter cy denoting the matching degree"of manuscript i for reviewer j, where iE ter a, is the certain number of reviewers that manuscript i should be assigned to: Parameter b, is the certain number of manuscripts that reviewer j should be as- signed to no more than; a given threshold T can be set as boundary of c to identify reviewers'qualification o The simplest version of RAP just distinguishes whether each reviewer and each anuscript can be matched or not, and represents the matching degree as a px matrix C=Ci(i=lp,j=lr), where cy is a binary parameter whose value equals l if there is overlap between reviewers expertise and manuscript's topic and no conflicts of interest, otherwise 0. As the binary ci is too general to present the grade of matching, discrete matching degree becomes a more popular method. A typical but exhausted way is requiring reviewers to rate their preference according the abstracts, with I for the lowest preference to 10 for the highest, sometimes p is used for the highest, where p is the total number of manuscripts. Once there is con flict of interest, cy is set to 0 [24]. Instead of rating for all the submissions, reviewers are usually required to score their expertise level on different disciplines(keywords)

722 F. Wang, B. Chen, and Z. Miao proposed a unified probabilistic framework for combining content-based and collabo￾rative IR by extending Hofmann’s [23] aspect model to incorporate the information source among scholar, manuscript and manuscript content. Semantic Web technology was also brought to collect data, represent researches’ expertise and co-authorship, and find relevant reviewers in a peer-review setting [6]. As the increase of internet resource, data mining can be used to identify relative reviewers within an existing expert pool by mining online information through search engines, potential external reviewers can also be found in the online academic community. In respond to the difficulty for a scientific community to agree upon taxonomy of keywords and maintain such a keyword database over time, Hettich and Pazzani [21] described the data mining method deployed at the U.S. NSF for assisting program directors in identifying reviewers for proposals. To the best of our knowledge, their paper is the only one work found in literature which combined IR and optimization to achieve RAP. 4 Reviewer Assignment Optimization Optimization on reviewer assignment which mainly focus on the theory, modeling and algorithm of assigning manuscripts to reviewers, can be viewed as an enhanced version of the Generalized Assignment Problem (GAP). However, most of these stud￾ies are not based on the matching degree evaluated by IR, but simply in sense of simi￾larity by the selected keywords. In making it easier to read, the notations used in the cited sources have, where necessary, been changed to try to keep consistent as those of traditional GAP. The RAP optimization we study is the following. We are given a set P = {1,..., p} of manuscripts and a set R = {1,...,r} of reviewers; and a parameter ij c denoting the “matching degree” of manuscript i for reviewer j , where i ∈ P and j ∈ R . Parame￾ter i a is the certain number of reviewers that manuscript i should be assigned to; Parameter bj is the certain number of manuscripts that reviewer j should be as￾signed to no more than; a given threshold T can be set as boundary of ij c to identify reviewers’ qualification. The simplest version of RAP just distinguishes whether each reviewer and each manuscript can be matched or not, and represents the matching degree as a p × r matrix ij C = c ( i = 1,..., p , j = 1,...,r ), where ij c is a binary parameter whose value equals 1 if there is overlap between reviewer’s expertise and manuscript’s topic and no conflicts of interest, otherwise 0. As the binary ij c is too general to present the grade of matching, discrete matching degree becomes a more popular method. A typical but exhausted way is requiring reviewers to rate their preference according to the abstracts, with 1 for the lowest preference to 10 for the highest, sometimes p is used for the highest, where p is the total number of manuscripts. Once there is con￾flict of interest, ij c is set to 0 [24]. Instead of rating for all the submissions, reviewers are usually required to score their expertise level on different disciplines (keywords)