《电子商务 E-business》阅读文献：A Multidimensional Paper Recommender Experiments and Evaluations

∽u=9◎ 2 A Multidimensional Paper Recommender Experiments and Evaluations uE Paper recommender systems in the e-learning domain must consider pedagogical factors, such as a paper's overall popularity and learner background nowledge- factors that are less important in commercial book or movie recommender systems. This article reports evaluations of a 6D paper recommender. Experimental results from a human subject study of learner preferences suggest that pedagogical factors help to overcome a serious cold start problem(not having enough papers or learners to start the recommender system) and help the system more appropriately support users as they learn. Tiffany Y Tang W hen information overload inten- collaborative filtering, construct groups Konkuk University sifies. users are overwhelmed of like-minded users with whom the by the information pouring target user shares similar interests and Gordon mccalla from various sources, including the makes recommendations based on an University of Saskatchewan Internet. Often, they're confused by analysis of them. which information they should con Making recommendations to learn ume that is, they find it difficult to ers differs from making recommenda pick the most appropriate information tions in many commercial domains, when the number of choices increases. in which only user likes and dislikes Recommender systems offer a feasible matter. A learner's overall impres solution to this problem. For example, if sion toward each paper depends not a user explicitly indicates a preference only on how interesting a paper is for action movies starring Sean Penn, but also on the degree that the paper a recommender system might suggest helps them meet their cognitive goals The Interpreter. In this case, the system This is consistent with Soo Young matches user preferences to movies' Rieh's observations of human users content features using a content-based information-seeking behaviors: "people other major recommendation approach, and cognitive authority ation quality filtering approach. Systems using an- make judgment of inforn 34 Published by the IEEE Computer Society 089-7801/09/$2500°2009IEE EEE INTERNET COMPUTING

Emerging E-Learning Technologies 34 Published by the IEEE Computer Society 1089-7801/09/$25.00 © 2009 IEEE IEEE INTERNET COMPUTING When information overload inten￾sifies, users are overwhelmed by the information pouring from various sources, including the Internet. Often, they’re confused by which information they should con - sume — that is, they find it difficult to pick the most appropriate information when the number of choices increases. Recommender systems offer a feasible solution to this problem. For example, if a user explicitly indicates a preference for action movies starring Sean Penn, a recommender system might suggest The Interpreter. In this case, the system matches user preferences to movies’ content features using a content-based filtering approach. Systems using an - other major recommendation approach, collaborative filtering, construct groups of like-minded users with whom the target user shares similar interests and makes recommendations based on an analysis of them. Making recommendations to learn - ers differs from making recommenda - tions in many commercial domains, in which only user likes and dislikes matter. A learner’s overall impres - sion toward each paper depends not only on how interesting a paper is, but also on the degree that the paper helps them meet their cognitive goals. This is consistent with Soo Young Rieh’s observations of human users’ information-seeking behaviors: “people make judgment of information quality and cognitive authority.” 1 Paper recommender systems in the e-learning domain must consider pedagogical factors, such as a paper’s overall popularity and learner background knowledge — factors that are less important in commercial book or movie recommender systems. This article reports evaluations of a 6D paper recommender. Experimental results from a human subject study of learner preferences suggest that pedagogical factors help to overcome a serious cold￾start problem (not having enough papers or learners to start the recommender system) and help the system more appropriately support users as they learn. Tiffany Y. Tang Konkuk University Gordon McCalla University of Saskatchewan A Multidimensional Paper Recommender Experiments and Evaluations

A Multidimensional Paper Recommender Table I Learner fe 3 L. How difficult is the paper to understand? Very difficult Difficult 2. Is the paper's content related to your job Not really Not at all 3. Is the paper interesting? Somewhat Not at al 4. Does the paper aid your understanding of the Very much Not really Not at all software engineering concepts and techniques learned in class? 5. Do you learn something new after reading lutely A little Not really Not at all pap 6. What is your overall rating of this paper? Very good 7. Will you recommen No classmates? 6 Existing research on recommending papers a learner be willing to tolerate a paper that learners doesn't consider pedagogical fac- isn't interesting? tors when making recommendations -that is, Will a learner be comfortable with a highly whether a recommended paper will enhance technical paper, even if it matches the learn- learning (see the "Related Work in Recom- er's interest or is required reading (for in- mender Systems"sidebar). To deal with this is- stance, a seminal paper on a topic)? sue, we proposed in earlier work the notion of recommending pedagogically appropriate pa- Due to space limitations, we only report our key pers .4Here, we extend our previous work by findings relevant to the first two questions determining the extra value of making peda gogically relevant recommendations. Learner Data Collection satisfaction is a complex function of learner Our study subjects were students enrolled in a characteristics and, as such, requires more master's program at the Hong Kong Polytechnic than simply matching a paper's topic against a University. All were registered in the "Software learner's interests. To study this claim, we look Engineering Concepts"course, whose curricu- at two factors: the significance of the pedagog- lum was designed primarily for mature or work ical factors in recommending a paper and the ing students from various backgrounds. A total ssociations among these factors that might in- of 40 part-time students attended the course. dicate their interactive relationships. We study The course curriculum included 22 papers as these factors in a human-subject experiment reading assignments, with a varying number and propose a recommendation algorithm ap- of papers assigned each week. Twenty-four stu- propriate to our findings dents agreed to participate in our experiment Empirical Study Learner profiles and feedback We performed an empirical study aimed at ex- We drew learner profiles from a questionnaire ploring the multidimensionality of paper recom- consisting of four categories: interest, knowl- mendation. In particular, we wanted to examine edge background, work experience, and learn he effect of considering pedagogical elements ing expectations. Students represented a pool in making recommendations. We addressed the of learners with work experience related to in following questions: formation technology, but not necessarily with computer science backgrounds. We used partic. Will learners be happy with papers that can ipants' learning expectations to determine what expand their knowledge (that is, teach them specific areas they wanted to pursue in their something new? E-learning's ultimate goal work, which the system could then use to make is fulfilling the user's knowledge needs. recommendations. After the students read each How important is learner interest in the rec- of the 22 assigned papers, we asked them to an- ommendation? For instance, how much will swer several questions, shown in Table 1 ULYJAUGUST 2009

JULY/AUGUST 2009 35 A Multidimensional Paper Recommender Existing research on recommending papers for learners2 doesn’t consider pedagogical fac￾tors when making recommendations — that is, whether a recommended paper will enhance learning (see the “Related Work in Recom￾mender Systems” sidebar). To deal with this is￾sue, we proposed in earlier work the notion of recommending pedagogically appropriate pa￾pers.3,4 Here, we extend our previous work by determining the extra value of making peda￾gogically relevant recommendations. Learner satisfaction is a complex function of learner characteristics and, as such, requires more than simply matching a paper’s topic against a learner’s interests. To study this claim, we look at two factors: the significance of the pedagog￾ical factors in recommending a paper and the associations among these factors that might in￾dicate their interactive relationships. We study these factors in a human-subject experiment and propose a recommendation algorithm ap￾propriate to our findings. Empirical Study We performed an empirical study aimed at ex￾ploring the multidimensionality of paper recom￾mendation. In particular, we wanted to examine the effect of considering pedagogical elements in making recommendations. We addressed the following questions: • Will learners be happy with papers that can expand their knowledge (that is, teach them something new)? E-learning’s ultimate goal is fulfilling the user’s knowledge needs. • How important is learner interest in the rec￾ommendation? For instance, how much will a learner be willing to tolerate a paper that isn’t interesting? • Will a learner be comfortable with a highly technical paper, even if it matches the learn￾er’s interest or is required reading (for in￾stance, a seminal paper on a topic)? Due to space limitations, we only report our key findings relevant to the first two questions. Data Collection Our study subjects were students enrolled in a master’s program at the Hong Kong Polytechnic University. All were registered in the “Software Engineering Concepts” course, whose curricu￾lum was designed primarily for mature or work￾ing students from various backgrounds. A total of 40 part-time students attended the course. The course curriculum included 22 papers as reading assignments, with a varying number of papers assigned each week. Twenty-four stu￾dents agreed to participate in our experiment. Learner Profiles and Feedback We drew learner profiles from a questionnaire consisting of four categories: interest, knowl￾edge background, work experience, and learn￾ing expectations. Students represented a pool of learners with work experience related to in￾formation technology, but not necessarily with computer science backgrounds. We used partic￾ipants’ learning expectations to determine what specific areas they wanted to pursue in their work, which the system could then use to make recommendations. After the students read each of the 22 assigned papers, we asked them to an￾swer several questions, shown in Table 1. Table 1. Learner feedback form. Question Response 4 3 2 1 1. How difficult is the paper to understand? Very difficult Difficult Easy Very easy 2. Is the paper’s content related to your job? Very Somewhat Not really Not at all 3. Is the paper interesting? Very Somewhat Not really Not at all 4. Does the paper aid your understanding of the software engineering concepts and techniques learned in class? Very much Somewhat Not really Not at all 5. Do you learn something new after reading this paper? Absolutely A little Not really Not at all 6. What is your overall rating of this paper? Very good Good Mediocre Bad 7. Will you recommend this paper to your classmates? Absolutely Maybe No

Emerging E-Learning Technologies teresting the paper is but also on the richness of knowledge that the learner has gained from reading it and its usefulness in helping the learner understand the course material Figure 1. Causal inference with partial correlation. When(a)rABc. The closeness of a learner's work experience effect, which we call a spurious to a paper's topic might also affect the learn explanation. When(b) TaB >abc>0, A partially affects B and er's overall ratings of that paper and the might affect C. We call this a partial explanation. likelihood of recommending it to others To validate our hypotheses, we must show that le used a Likert 4-scale rating for the par- ratings on the value-added or aid-learning fac ticipants' answers to all but the last question. tors (question 4) affect ratings on overall or Several questions are related to a paper's peda- peer recommendation independently from how gogical values the degree of difficulty to un- interesting the paper is (3). We performed four derstand (question 1), the users job-relatedness analyses: partial correlation, structural equa- (2), the amount of value added (5), and degree of tion modeling, principal components analysi peer recommendation (7). Because the candidate (PCA), and partial least squares regression(PLS) papers are mainly from popular technical maga- These statistical analysis methods are particu zines, even learners with no deep mathematical larly valuable to social scientists for probing knowledge have little difficulty understanding the interactivity among variables. Due to space them. We expected the most difference among limitations, we report only our main findings learners for the papers' value added, degree of from partial correlation, PCA, and PLs peer recommendation, and overall rating Partial Correlation results Correlation Analysis Partial correlation is often used for modeling Our major goal in analyzing the data collected causality using three or four variables. Let TAB.C through the study is to determine the factors be the Pearson correlation of variables A and B, that rank a paper high in terms of its pedagogi- controlling for variable C; and Tab be the Pear- cal benefits. Were looking for important inter- son correlation of variables A and B If TAB.C actions among the pedagogical variables that TAB, we infer that the control variable C has no might help accurately rank a paper for a learner. effect. If TAB.c approaches 0, TAB is spurious (the To achieve this, we don't use traditional evalu- correlation is spurious)-that is, no direct caus- tion approaches such as mean absolute error, al link exists between a and b(see Figure 1a) which examine how close the recommender Rather, either C affects A and b (anteceding) systems predicted ratings are to the true user or A affects C, which affects B (intervening ). If ratings, or receiver operating characteristics, AB > TAB.c>0, we have partial explanation (see which measure how well users receive the rec- Figure 1b). In this case, a partially affects b re ommended items. Instead, we use statistical gardless of whether it affects (or is affected by) methods to sort out alternative explanations C Table 2 shows our partial correlation compu- for relations between variables and therefore to tations for the data collected in the study. further our understanding of the relationship of We divided the analysis into three groups various pedagogical factors to the ranking of (I, Il, and Ill), based on different assumptions papers appropriate for particular learners. We about what were correlating make the following hypotheses In Group I, we check TaB and TaBc for A value_added, B= overall or peer_rec, A paper's overall rating might not only de- = interest. The results of TAB are 0. 4798 and pend on how interesting it is but also on the 0.4335 for B overall and B= peer_rec,re- richness of knowledge that the learner gains spectively. After introducing interest as a rom reading it and its usefulness in help- control, the correlations decrease to 0. 3539 and ing a learner understand the course subject 0.3017, respectively; hence, for this group, TAb matter. A learners intent in recommending a In Group Il, we check TAB and TAB to others might depend not only on ho aid_learning. EEE INTERNET COMPUTING

Emerging E-Learning Technologies 36 www.computer.org/internet/ IEEE INTERNET COMPUTING We used a Likert 4-scale rating for the par￾ticipants’ answers to all but the last question. Several questions are related to a paper’s peda￾gogical values — the degree of difficulty to un￾derstand (question 1), the user’s job-relatedness (2), the amount of value added (5), and degree of peer recommendation (7). Because the candidate papers are mainly from popular technical maga￾zines, even learners with no deep mathematical knowledge have little difficulty understanding them. We expected the most difference among learners for the papers’ value added, degree of peer recommendation, and overall rating. Correlation Analysis Our major goal in analyzing the data collected through the study is to determine the factors that rank a paper high in terms of its pedagogi￾cal benefits. We’re looking for important inter￾actions among the pedagogical variables that might help accurately rank a paper for a learner. To achieve this, we don’t use traditional evalu￾ation approaches such as mean absolute error, which examine how close the recommender system’s predicted ratings are to the true user ratings, or receiver operating characteristics, which measure how well users receive the rec￾ommended items. Instead, we use statistical methods to sort out alternative explanations for relations between variables and therefore to further our understanding of the relationship of various pedagogical factors to the ranking of papers appropriate for particular learners. We make the following hypotheses: • A paper’s overall rating might not only de￾pend on how interesting it is but also on the richness of knowledge that the learner gains from reading it and its usefulness in help￾ing a learner understand the course subject matter. • A learner’s intent in recommending a paper to others might depend not only on how in￾teresting the paper is but also on the richness of knowledge that the learner has gained from reading it and its usefulness in helping the learner understand the course material. • The closeness of a learner’s work experience to a paper’s topic might also affect the learn￾er’s overall ratings of that paper and the likelihood of recommending it to others. To validate our hypotheses, we must show that ratings on the value-added or aid-learning fac￾tors (question 4) affect ratings on overall or peer recommendation independently from how interesting the paper is (3). We performed four analyses: partial correlation, structural equa￾tion modeling, principal components analysis (PCA), and partial least squares regression (PLS). These statistical analysis methods are particu￾larly valuable to social scientists for probing the interactivity among variables. Due to space limitations, we report only our main findings from partial correlation, PCA, and PLS. Partial Correlation Results Partial correlation is often used for modeling causality using three or four variables. Let rAB.C be the Pearson correlation of variables A and B, controlling for variable C; and rAB be the Pear￾son correlation of variables A and B. If rAB.C = rAB, we infer that the control variable C has no effect. If rAB.C approaches 0, rAB is spurious (the correlation is spurious) — that is, no direct caus￾al link exists between A and B (see Figure 1a). Rather, either C affects A and B (anteceding) or A affects C, which affects B (intervening). If rAB > rAB.C > 0, we have partial explanation (see Figure 1b). In this case, A partially affects B re￾gardless of whether it affects (or is affected by) C. Table 2 shows our partial correlation compu￾tations for the data collected in the study. We divided the analysis into three groups (I, II, and III), based on different assumptions about what we’re correlating. In Group I, we check rAB and rAB.C for A = value_added, B = overall or peer_rec, and C = interest. The results of rAB are 0.4798 and 0.4335 for B = overall and B = peer_rec, re￾spectively. After introducing interest as a control, the correlations decrease to 0.3539 and 0.3017, respectively; hence, for this group, rAB > rAB.C > 0. In Group II, we check rAB and rAB.C for A = aid_learning. C A B (a) C A B C A B (b) C A B Figure 1. Causal inference with partial correlation. When (a) rAB.C = 0, the control variable C has no effect, which we call a spurious explanation. When (b) rAB > rAB.C > 0, A partially affects B and might affect C. We call this a partial explanation

A Multidimensional Paper Recommender Table 2. Results from the partial correlation analysis. Pearson partial correlation(TAB.c) Variables C(co A B(overall) B(peer_rec) 0.4798 04335 teres Value added 30l7 NIA Aid learning 0.4242 0.3740 Interest Aid_learning 0.3038 0.2469 Interest 0.6046 0.5574 Value added Interest 0.5282 04780 Aid_learning Interest 0.5456 0.4975 In Group Ill, we check TAB and T AB C for the reverse causality that is, interest is affected by value_added or aid_learning. In fact, TAb> TAB.c>0 for all groups, or the results favor a par tial explanation model. In other words, to some degree, value_added and aid_learning affect 04 verall and peer_rec ratings independently 0 from interest. Whether the model is still valid in the presence of other variables (such as dif- Figure 2. The variable-importance-in-the-projection(VP)index of ficulty or job_related)is unclear. omponents /(left bar: overall) and 2 (right bar: peer_rec) from partial least squares regression(PLS). Any independent Principal Components Regression and variable with a small VP(that is, less than 0.8)could be considered Partial Least Squares Regression Results unimportant and hence can be dropped from the model.6 Principal components regression (PCR)com- bines PCA and linear regression. PCA trans- forms observations from a p-dimensional space decomposes explanatory and depende original dimensions. The resulting dimensions between them, OnS to a q-dimensional space(q s p) while conserv- ables, with the constraint that these ing as much information as possible from the explain as much as possible of the cov are noncorrelated weighted components that Ne set the stopping criteria for both PCR are linear combinations of the original vari- and PLs as the point at which they find at most ibles. The weights are usually represented by two components. The regression therefore ex- eigenvalues produced during transformation. cludes other components. We used Addinsoft's High-eigenvalue components contain the most XLSTAT 2007 to perform both PCR and PLS researchers to analyze the associations between aid_learning, and value_ added as explan, information from the original data, allowing We used difficulty, job_related, interest, he original variables. Meanwhile, by removing tory variables; and overall and peer_rec as low-eigenvalue components, we can also sim- dependent variables. The PCR model uses 61 plify the regression model. If an explanatory percent variability of the original explanatory variable is redundant (for example, collinear data. This value is low (the suggested variabl- with other variables), it will vanish during di- ity in PCR is at least 80 percent in XLSTAT) mensional reduction by PCR. Our test checks We restrict our model to a low variability to whether value_added or aid_learning impact verify the survivability of value_added and the overall rating, peer_rec, or both. id_learning as explanatory variables in the PLS also uses building noncorrelated model. Results support these variables'surviv- components but differs from PCR in that it con- ability in explaining overall and peer_rec siders regression accuracy when selecting com- ratings. In addition, the variable-importance- ponents in regression. The selected components in-the-projection index of both variables from arent necessarily those with the highest eigen- PLS are higher than the critical value 0.8, values, but those that explain the most inde- leading us to strongly believe that they con- pendent variables. As such, PLS simultaneously tribute significantly to the model, as Figure 2 ULYJAUGUST 2009

JULY/AUGUST 2009 37 A Multidimensional Paper Recommender In Group III, we check rAB and rAB.C for the reverse causality — that is, interest is affected by value_added or aid_learning. In fact, rAB > rAB.C > 0 for all groups, or the results favor a par￾tial explanation model. In other words, to some degree, value_added and aid_learning affect overall and peer_rec ratings independently from interest. Whether the model is still valid in the presence of other variables (such as dif￾ficulty or job_related) is unclear. Principal Components Regression and Partial Least Squares Regression Results Principal components regression (PCR) com￾bines PCA and linear regression. PCA trans￾forms observations from a p-dimensional space to a q-dimensional space (q ≤ p) while conserv￾ing as much information as possible from the original dimensions. The resulting dimensions are noncorrelated weighted components that are linear combinations of the original vari￾ables. The weights are usually represented by eigenvalues produced during transformation. High-eigenvalue components contain the most information from the original data,5 allowing researchers to analyze the associations between the original variables. Meanwhile, by removing low-eigenvalue components, we can also sim￾plify the regression model. If an explanatory variable is redundant (for example, collinear with other variables), it will vanish during di￾mensional reduction by PCR. Our test checks whether value_added or aid_learning impact the overall rating, peer_rec, or both. PLS also uses PCA in building noncorrelated components but differs from PCR in that it con￾siders regression accuracy when selecting com￾ponents in regression. The selected components aren’t necessarily those with the highest eigen￾values, but those that explain the most inde￾pendent variables. As such, PLS simultaneously decomposes explanatory and dependent vari￾ables, with the constraint that these variables explain as much as possible of the covariance between them. We set the stopping criteria for both PCR and PLS as the point at which they find at most two components. The regression therefore ex￾cludes other components. We used Addinsoft’s XLSTAT 2007 to perform both PCR and PLS. We used difficulty, job_related, interest, aid_learning, and value_added as explana￾tory variables; and overall and peer_rec as dependent variables. The PCR model uses 61.1 percent variability of the original explanatory data. This value is low (the suggested variabil￾ity in PCR is at least 80 percent in XLSTAT). We restrict our model to a low variability to verify the survivability of value_added and aid_learning as explanatory variables in the model. Results support these variables’ surviv￾ability in explaining overall and peer_rec ratings. In addition, the variable-importance￾in-the-projection index of both variables from PLS are higher than the critical value 0.8, leading us to strongly believe that they con￾tribute significantly to the model,6 as Figure 2 Table 2. Results from the partial correlation analysis. Pearson partial correlation (rAB.C) Variables C (control) A B (overall) B (peer_rec) Group I N/A Value_added 0.4798 0.4335 Interest Value_added 0.3539 0.3017 Group II N/A Aid_learning 0.4242 0.3740 Interest Aid_learning 0.3038 0.2469 Group III N/A Interest 0.6046 0.5574 Value_added Interest 0.5282 0.4780 Aid_learning Interest 0.5456 0.4975 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Interesting Value_added Aid_learning Difficulty Job_related VIP Figure 2. The variable-importance-in-the-projection (VIP) index of components 1 (left bar: overall) and 2 (right bar: peer_rec) from partial least squares regression (PLS). Any independent variable with a small VIP (that is, less than 0.8) could be considered unimportant and hence can be dropped from the model.6

Emerging E-Learning Technologies overall,Walue_added,W)0.0,0.0.0.05.09040.09400.03.09600202,(.0.0) Popularity(w,) (,0),(,0.5),(l,)} W2DStdModel 0.5,1,3,5,10} Number of neighbors demonstrates. In other words, a papers value- +W added and aid-learning factors are important in making recommendations where n is the number of neighbors IBI and W is the weight of a paper's popularity F Proposed Paper Recommender Using the ranking r,, we can find the best Our paper recommender uses collaborative filter- papers to recommend. ng to find papers for a learner that have proven Our paper recommendation system there- to be pedagogically relevant for similar learners. fore uses six elements: overall performance, Several factors contribute to determining such value added, peer recommendation, F, learner similarity. First, we consider three factors as a interest, and learner background knowledge basis for measuring a pair of learners' closeness: The 6D collaborative-filtering computation is overall, value added, and peer recommendation. more complex than other collaborative-filter- Because each paper has three different ratings, ing-based techniques; however, we believe that we can obtain three different Pearson correlations under certain circumstances, it's necessary and for each pair of learners. Suppose Pa (a, b) is the beneficial. (In addition to 6D collaborative fil Pearson correlation based on the rating ra on di- tering, we've studied 3D and 4D computations mension d. Then, we can combine those three cor- and compared performances. relations into a weighted-sum Pearson correlation as P3p (a, b)=Overall Overal(a, b)+wvalue added Pval. Evaluation added(a, b)+Wpeer_ree Ppeer-recla, b), where overall+ To test our 6D recommender system, we used Wvalue_ added+Peer._ree=1(w= weight e Through computation, we find a group of in the empirical study described earlier. We've rners similar to a given target learner. explored a vast number of weight combinations Next, we compute the 2D Pearson correlation and neighborhood sizes in the 6D collaborative- between learners based on their student mod- filtering technique to find the factors that would els. That is, we compute the aggregated Pearson lead to recommendations that best matched a correlation between student interest and back- student's actual experience ground knowledge as P2Dstd Model(a, b ) Interest We found that the weighting combina- Interest(a, b)+ wbkgrKnowledge Pbkgrknowledgela, b). tions overall, value_added, and peer_rec Because we have various weights on combin- were most important. However, they benefit the ing Pearson correlations, we can tune them to recommendation only when the sum of value study each factor's relative importance. We then _added and peer_rec is less than 0. 1. The combine this result with a 3D Pearson correla- overall rating is still the most important factor. tion from coated papers: Psp(a, b)= P3p(a, b ) le set the number of neighbors to five and W2p P2DstdModella, ) From Pspa, b), we can iden- 10, although we also tested other values. We tify the best n neighbors for a target user. After were also eager to see which factors better boost that, we use the following formula to calculate recommendation performance. Table 3 summa- each paper's aggregate rating rizes the sets of variables in 6d collaborative lItering. The weight reflects the corresponding =∑D(a,b)hk variables value Lastly, we combine this rating with each pa- Evaluation Protocols per's average rating (that is, the paper's popular- For each target learner, we randomly assigned ity )to obtain a 6D collaborative-filtering-based 30 combinations of coated papers to find neigh rating, as follows: bors, who later recommend one or five papers EEE INTERNET COMPUTING

Emerging E-Learning Technologies 38 www.computer.org/internet/ IEEE INTERNET COMPUTING demonstrates. In other words, a paper’s value￾added and aid-learning factors are important in making recommendations. Proposed Paper Recommender Our paper recommender uses collaborative filter￾ing to find papers for a learner that have proven to be pedagogically relevant for similar learners. Several factors contribute to determining such similarity. First, we consider three factors as a basis for measuring a pair of learners’ closeness: overall, value added, and peer recommendation. Because each paper has three different ratings, we can obtain three different Pearson correlations for each pair of learners. Suppose Pd (a, b) is the Pearson correlation based on the rating rd on di￾mension d. Then, we can combine those three cor￾relations into a weighted-sum Pearson correlation as P3D (a,b) = woverall Poverall(a,b) + wvalue_added Pval￾ue_added(a,b) + wpeer_rec Ppeer_rec(a,b), where woverall + wvalue_added + wpeer_rec = 1 (w = weight). Through computation, we find a group of learners similar to a given target learner. Next, we compute the 2D Pearson correlation between learners based on their student mod￾els. That is, we compute the aggregated Pearson correlation between student interest and back￾ground knowledge as P2DStdModel(a,b) = winterest Pinterest(a,b) + wbkgrKnowledge PbkgrKnowledge(a,b). Because we have various weights on combin￾ing Pearson correlations, we can tune them to study each factor’s relative importance. We then combine this result with a 3D Pearson correla￾tion from corated papers: P5D(a,b) = P3D(a,b) + w2D P2DStdModel(a,b). From P5D(a,b), we can iden￾tify the best n neighbors for a target user. After that, we use the following formula to calculate each paper’s aggregate rating: r P a b r k D D b k B 5 = 5 ∑ ( ) , , . Lastly, we combine this rating with each pa￾per’s average rating (that is, the paper’s popular￾ity ) to obtain a 6D collaborative-filtering-based rating, as follows: r r w nr k D k D r 6 5 = +   , where n is the number of neighbors = |B| and wr is the weight of a paper’s popularity r. Using the ranking r k 6D , we can find the best papers to recommend. Our paper recommendation system there￾fore uses six elements: overall performance, value added, peer recommendation, r, learner interest, and learner background knowledge. The 6D collaborative-filtering computation is more complex than other collaborative-filter￾ing-based techniques; however, we believe that under certain circumstances, it’s necessary and beneficial. (In addition to 6D collaborative fil￾tering, we’ve studied 3D and 4D computations and compared performances.) Evaluation To test our 6D recommender system, we used it to make recommendations for the students in the empirical study described earlier. We’ve explored a vast number of weight combinations and neighborhood sizes in the 6D collaborative￾filtering technique to find the factors that would lead to recommendations that best matched a student’s actual experience. We found that the weighting combina￾tions overall, value_added, and peer_rec were most important. However, they benefit the recommendation only when the sum of value _added and peer_rec is less than 0.1. The overall rating is still the most important factor. We set the number of neighbors to five and 10, although we also tested other values. We were also eager to see which factors better boost recommendation performance. Table 3 summa￾rizes the sets of variables in 6D collaborative filtering. The weight reflects the corresponding variable’s value. Evaluation Protocols For each target learner, we randomly assigned 30 combinations of corated papers to find neigh￾bors, who later recommend one or five papers. Table 3. Experimental variables. Dimensions Weighting values (woverall, wvalue_added, wpeer_rec) {(0.8, 0.1, 0.1), (0.9, 0.05, 0.05), (0.92, 0.04, 0.04), (0.94, 0.03, 0.03), (0.96, 0.02, 0.02), (1, 0, 0)} Popularity ( wr ) {1, 5} (winterest, wbkgrKnowledge) {(1, 0), (1, 0.5), (1, 1)} w2DStdModel {0.5, 1, 3, 5, 10} Number of neighbors {5, 10}