The current issue and full text archive of this journal is available at www.emeraldinsight.com/1468-4527.htm DIR OTHER ARTICLE 33,3 An opinion-based decision model for recommender systems 84 Sea woo kim Refereed article received Division of Information and Communication engineering Korea Advanced institute of Science and Technology, Seoul, South Korea Approved for publication 21 October 2008 Chin-Wan Chung Department of computer Science Korea Advanced institute of science and Technology, Seoul, South Korea, and Daeeun kim School of electrical and Electronic Engineering, Yonsei university Seoul. South Korea Purpose A good recommender system helps users find items of interest on the web and can provide commendations based on user preferences. In contrast to automatic technology.generate recommender systems, this paper aims to use dynamic expert groups that are automatically formed o recommend domain-specific documents for general users. In addition, it aims to test several effectiveness measures of rank order to determine if the top-ranked lists recommended by the experts were reliable Design/methodologylapproach-In the approach, expert groups evaluate web documents to provide a recommender system for general users. The authority and make-up of the expert group are adjusted through user feedback. The system also uses various measures to gauge the difference between the opinions of experts and those of general users to improve the evaluation effectiveness. Findings- The proposed system is efficient when there is major support from experts and general users. The recommender system is especially effective where there is a limited amount of evaluation data from general users. Originality/value- This study of how to effectively recommend web documents to users based on the opinions of experts. Simulation results were provided to show the effectiveness of the dynamic expe for recommender systems. Keywords Information retrieval, Skills, Worldwide web Paper type Research paper Introduction Emerald The development of recommender systems as a means of information retrieval has emerged as an important issue of the internet, and has drawn attention both from academics and the commercial sector Online Information Review This research was supported by the Ministry of Knowledge Economy, Korea under th DEmeald Group Publishing Limited Information Technology Research Center support programme supervised by the Institute of DoI 101108/1468152091090970 Information Technology Advancement(grant number IlTA-2008-C1090-0801-0031)
OTHER ARTICLE An opinion-based decision model for recommender systems Sea Woo Kim Division of Information and Communication Engineering, Korea Advanced Institute of Science and Technology, Seoul, South Korea Chin-Wan Chung Department of Computer Science, Korea Advanced Institute of Science and Technology, Seoul, South Korea, and DaeEun Kim School of Electrical and Electronic Engineering, Yonsei University, Seoul, South Korea Abstract Purpose – A good recommender system helps users find items of interest on the web and can provide recommendations based on user preferences. In contrast to automatic technology-generated recommender systems, this paper aims to use dynamic expert groups that are automatically formed to recommend domain-specific documents for general users. In addition, it aims to test several effectiveness measures of rank order to determine if the top-ranked lists recommended by the experts were reliable. Design/methodology/approach – In the approach, expert groups evaluate web documents to provide a recommender system for general users. The authority and make-up of the expert group are adjusted through user feedback. The system also uses various measures to gauge the difference between the opinions of experts and those of general users to improve the evaluation effectiveness. Findings – The proposed system is efficient when there is major support from experts and general users. The recommender system is especially effective where there is a limited amount of evaluation data from general users. Originality/value – This is an original study of how to effectively recommend web documents to users based on the opinions of human experts. Simulation results were provided to show the effectiveness of the dynamic expert group for recommender systems. Keywords Information retrieval, Skills, Worldwide web Paper type Research paper Introduction The development of recommender systems as a means of information retrieval has emerged as an important issue of the internet, and has drawn attention both from academics and the commercial sector. The current issue and full text archive of this journal is available at www.emeraldinsight.com/1468-4527.htm This research was supported by the Ministry of Knowledge Economy, Korea under the Information Technology Research Center support programme supervised by the Institute of Information Technology Advancement (grant number IITA-2008-C1090-0801-0031). OIR 33,3 584 Refereed article received 1 April 2008 Approved for publication 21 October 2008 Online Information Review Vol. 33 No. 3, 2009 pp. 584-602 q Emerald Group Publishing Limited 1468-4527 DOI 10.1108/14684520910969970
An example of the use of such systems is to recommend new products or items of An opinion- interest to online customers, using customer preferences. When customers have no based decision personal experience of an item or class of items, they are often interested in retrieving nformation about the items or products that many others have ordered or used. However, model many recommender systems have focused on retrieving information considering the preferences of just one or a few customers, and sometimes there may be no information about user preferences for a recommender system to draw on Recommender systems can 585 also be applied to retrieve relevant web documents. Web documents need users evaluations in order to be recommendable. A simple citation search for a relevant document may not reflect well the crucial point of a document. We need a different approach to recommender systems In our research we have explored a method in which human agents collect useful information and provide it to eneral users. a group of human users or experts can cooperate to determine whether a web document includes useful information and rank the web documents in order Such information can be made available to general users as recommendations. The feedback of the users can reorganise the group and refine the knowledge level that a group of human experts provides. This kind of adaptive organisation and feedback loop will give users access to expert knowledge in a particular domain, and will have a filtering effect on biased opinions from just a few people. Information retrieval systems often give numeric scores to documents and then rank them based on the scores in order to make recommendations to users there have been several approaches to the information retrieval system, the most common of which are the vector space model, the probabilistic model and the inference network model. In the vector space model, a document is represented by a vector of terms, that is, words and phrases (alton et al, 1975). The model calculates the similarity between a query and a document. The angle between the query vector and the document vector can be measured using the cosine property(the dot product of two vectors is involved with the cosine angle). More similar vectors will have a numeric cosine value close to 1. The probabilistic model estimates the probability of the relevance of a document to a query robertson, 1977) and documents can be ranked based on the relevance probability. In the inference network model, an inference process is applied to model the information retrieval turtle and croft, 1991), where a document instantiates a term with a certain strength, and it accumulates the credit from multiple terms to assign a numeric score to the document. Then the strength of instantiation is taken as the weight of the term in the document. The above scoring methods can assist in the automatic evaluation of documents But while this kind of numeric assignment can give a rough evaluation or coarse information retrieval, in many cases it cannot provide accurate information about given documents Many common web searches retrieve a very small number of relevant documents. Topic distillation is a special kind of topical relevance search where the user wishes to find a few key websites rather than every relevant webpage. Because these types of searches are so common, web search evaluations have come to focus on tasks where there are a very few relevant documents. Evaluations with just a few relevant documents pose special challenges for current metrics (Soboroff, 2006). The development of intelligent information retrieval techniques has large impact potential in many domains(fern et al, 2007)
An example of the use of such systems is to recommend new products or items of interest to online customers, using customer preferences. When customers have no personal experience of an item or class of items, they are often interested in retrieving information about the items or products that many others have ordered or used. However, many recommender systems have focused on retrieving information considering the preferences of just one or a few customers, and sometimes there may be no information about user preferences for a recommender system to draw on. Recommender systems can also be applied to retrieve relevant web documents. Web documents need many users’ evaluations in order to be recommendable. A simple citation search for a relevant document may not reflect well the crucial point of a document. We need a different approach to recommender systems. In our research we have explored a method in which human agents collect useful information and provide it to general users. A group of human users or experts can cooperate to determine whether a web document includes useful information and rank the web documents in order. Such information can be made available to general users as recommendations. The feedback of the users can reorganise the group and refine the knowledge level that a group of human experts provides. This kind of adaptive organisation and feedback loop will give users access to expert knowledge in a particular domain, and will have a filtering effect on biased opinions from just a few people. Information retrieval systems often give numeric scores to documents and then rank them based on the scores in order to make recommendations to users. There have been several approaches to the information retrieval system, the most common of which are the vector space model, the probabilistic model and the inference network model. In the vector space model, a document is represented by a vector of terms, that is, words and phrases (Salton et al., 1975). The model calculates the similarity between a query and a document. The angle between the query vector and the document vector can be measured using the cosine property (the dot product of two vectors is involved with the cosine angle). More similar vectors will have a numeric cosine value close to 1. The probabilistic model estimates the probability of the relevance of a document to a query (Robertson, 1977) and documents can be ranked based on the relevance probability. In the inference network model, an inference process is applied to model the information retrieval (Turtle and Croft, 1991), where a document instantiates a term with a certain strength, and it accumulates the credit from multiple terms to assign a numeric score to the document. Then the strength of instantiation is taken as the weight of the term in the document. The above scoring methods can assist in the automatic evaluation of documents. But while this kind of numeric assignment can give a rough evaluation or coarse information retrieval, in many cases it cannot provide accurate information about given documents. Many common web searches retrieve a very small number of relevant documents. Topic distillation is a special kind of topical relevance search where the user wishes to find a few key websites rather than every relevant webpage. Because these types of searches are so common, web search evaluations have come to focus on tasks where there are a very few relevant documents. Evaluations with just a few relevant documents pose special challenges for current metrics (Soboroff, 2006). The development of intelligent information retrieval techniques has large impact potential in many domains (Fern et al., 2007). An opinionbased decision model 585
DIR ecommender systems, as one information retrieval technique, can be broadly 33,3 categorised into content-based and collaborative filtering systems(hill et al, 1995 Resnick et al, 1994; Shardanand and Maes, 1995; Soboroff et al, 1999 ) Content-based filtering methods use textual descriptions of documents or items to be recommended. A users profile is associated with the content of the documents that the user has already rated. The features of documents are extracted by information retrieval, pattern 86 recognition or machine learning techniques. Then the content-based system recommends documents that match the users profile delgado et al, 1998; Soborof et al, 1999). In contrast, collaborative filtering systems are based on user ratings rather than the features of the documents(Breese et al, 1998; Soboroff et al, 1999: Shardanand and Maes, 1995). These systems predict the ratings of a user for given documents or items, depending on the ratings of other users with similar preferences to the user Collaborative filtering systems, such as groupLens (Resnick et al, 1994; Konstan et al. 1997), can be part of recommender systems for online shopping sites. They recommend items to users, using the history of products that similar users have ordered or have viewed Most recommender systems use analysis of the users preferences. Such systems require the user to judge many items in order to obtain the users preferences. In general, many online customers or users are interested in other users' opinions or ratings about items that belong to a certain category For instance, many e-commerce customers like to see the top-ranked lists of rating scores of many users for retail items in order to help them make a purchase decision. However, recommender systems still have difficulty providing relevant rating information before they receive a large number of user evaluations or feedbacks In this paper, we provide a new method for evaluating web documents using a representative board of human agents(an "expert group"). This is different from automatic recommender systems with software agents or feature extractions. We suggest that dynamic expert groups should be created from among users to evaluate domain-specific documents for webpage ranking, and that the group members should have dynamic authority weights depending on the performance of their ranking evaluations. This method will be quite effective in recommending web documents or items that many users have not already evaluated -in such cases it is difficult for automatic recommender services to provide effective recommendations. Because in our approach users with expertise in a domain category evaluate the documents, it is not feasible to replace human agents with intelligent software agents Our recommender system with dynamic expert groups may be extended to challenge search engine designs and image retrieval problems. Many search engines find relevant information and its importance by applying automatic citation analysis to the general subject of a query. The hypertext connectivity of web documents has been a good measure for automatic web citation analysis. This method works on the assumption that a webpage that is cited many times is popular and important. Many automatic page-ranking systems have used this citation metric to decide the relative importance of web documents. The IBM HITS system maintains a hub and an authority score for every document(Kleinberg, 1998). A method called PageRank computes a ranking for every web document based on a web connectivity graph ( brin and Page, 1998) with the random walk traversal. It also considers the relative
Recommender systems, as one information retrieval technique, can be broadly categorised into content-based and collaborative filtering systems (Hill et al., 1995; Resnick et al., 1994; Shardanand and Maes, 1995; Soboroff et al., 1999). Content-based filtering methods use textual descriptions of documents or items to be recommended. A user’s profile is associated with the content of the documents that the user has already rated. The features of documents are extracted by information retrieval, pattern recognition or machine learning techniques. Then the content-based system recommends documents that match the user’s profile (Delgado et al., 1998; Soboroff et al., 1999). In contrast, collaborative filtering systems are based on user ratings rather than the features of the documents (Breese et al., 1998; Soboroff et al., 1999: Shardanand and Maes, 1995). These systems predict the ratings of a user for given documents or items, depending on the ratings of other users with similar preferences to the user. Collaborative filtering systems, such as GroupLens (Resnick et al., 1994; Konstan et al., 1997), can be part of recommender systems for online shopping sites. They recommend items to users, using the history of products that similar users have ordered or have viewed. Most recommender systems use analysis of the user’s preferences. Such systems require the user to judge many items in order to obtain the user’s preferences. In general, many online customers or users are interested in other users’ opinions or ratings about items that belong to a certain category. For instance, many e-commerce customers like to see the top-ranked lists of rating scores of many users for retail items in order to help them make a purchase decision. However, recommender systems still have difficulty providing relevant rating information before they receive a large number of user evaluations or feedbacks. In this paper, we provide a new method for evaluating web documents using a representative board of human agents (an “expert group”). This is different from automatic recommender systems with software agents or feature extractions. We suggest that dynamic expert groups should be created from among users to evaluate domain-specific documents for webpage ranking, and that the group members should have dynamic authority weights depending on the performance of their ranking evaluations. This method will be quite effective in recommending web documents or items that many users have not already evaluated – in such cases it is difficult for automatic recommender services to provide effective recommendations. Because in our approach users with expertise in a domain category evaluate the documents, it is not feasible to replace human agents with intelligent software agents. Our recommender system with dynamic expert groups may be extended to challenge search engine designs and image retrieval problems. Many search engines find relevant information and its importance by applying automatic citation analysis to the general subject of a query. The hypertext connectivity of web documents has been a good measure for automatic web citation analysis. This method works on the assumption that a webpage that is cited many times is popular and important. Many automatic page-ranking systems have used this citation metric to decide the relative importance of web documents. The IBM HITS system maintains a hub and an authority score for every document (Kleinberg, 1998). A method called PageRank computes a ranking for every web document based on a web connectivity graph (Brin and Page, 1998) with the random walk traversal. It also considers the relative OIR 33,3 586
importance by checking the rank of documents a document is ranked as highly An opinion- important when the document has backlinks from documents with high authority, based decision such as the yahoo homepage. However, automatic citation analysis is limited in that it does not reflect well th model importance of a document from a human perspective. There are many cases where simple citation counting does not reflect our commonsense concept of importance ( Brin and Page, 1998). This research addresses this problem by exploring a method of 587 ranking based on human interactions, where a pool of expert human agents are used to evaluate web documents and their authority is dynamically determined through user feedback on their performance Rocchio(1971) proposed relevance feedback for query modification, where users judge the relevance of a document for a query and leave feedback. The system then updates the query based on the feedback. This has been shown to be quite effective in query modification. Following this idea, we apply the relevance feedback of users to the ranked documents provided by the expert group. The feedback information will modify the authority weight of the expert group members. As a result, the decisions of the expert group will reflect the feedback of users as time passes. In this paper, we suggest a novel recommender system based on human interactions. All the key decisions follow human opinions from a specialised or expert"group, so more reasonable recommendations can be made available in situations that are vague because few users have evaluated an item. Automatic selection or ejection of expert members based on their performance can be used to maintain the expertise of the group. The relevant documents provided by the expert group are sorted in rank order. To check the effectiveness of the system, we have developed several effectiveness measures based on rank order. In this paper we validate our approach with simulations of user feedback and expert group reorganisation, and evaluate the results using the new effectiveness measures. Our preliminary work was published in conference proceedings(Kim and Kim, 2001; Kim and Chung, 2001) Proposed method Dynamic authority weights of experts We define a group of people with high authority and much expertise in a special field as an expert group Figure 1 shows a framework for a search engine with our recommender system. a meta-search engine is used to collect good web documents from the conventional search engines(e.g. Yahoo, Alta Vista, Excite and InfoSeek). The addresses of the documents cited in the search engines are stored in the document database. Also recorded for each web document are details of how many search engines in the meta-search engine referred to the document, and how many times online users had accessed the web document using the search engine For every category there is a list of top-ranked documents rated by an expert group which are sorted by score. Authoritative webpages are determined by human expert group members. The experts examine the content of candidate webpages that are highly referenced among web documents or have been accessed by many users. The method of employing an expert group is based on the idea that for a given decision task requiring expert knowledge, many experts may be better than one if their individual
importance by checking the rank of documents – a document is ranked as highly important when the document has backlinks from documents with high authority, such as the Yahoo homepage. However, automatic citation analysis is limited in that it does not reflect well the importance of a document from a human perspective. There are many cases where simple citation counting does not reflect our commonsense concept of importance (Brin and Page, 1998). This research addresses this problem by exploring a method of ranking based on human interactions, where a pool of expert human agents are used to evaluate web documents and their authority is dynamically determined through user feedback on their performance. Rocchio (1971) proposed relevance feedback for query modification, where users judge the relevance of a document for a query and leave feedback. The system then updates the query based on the feedback. This has been shown to be quite effective in query modification. Following this idea, we apply the relevance feedback of users to the ranked documents provided by the expert group. The feedback information will modify the authority weight of the expert group members. As a result, the decisions of the expert group will reflect the feedback of users as time passes. In this paper, we suggest a novel recommender system based on human interactions. All the key decisions follow human opinions from a specialised or “expert” group, so more reasonable recommendations can be made available in situations that are vague because few users have evaluated an item. Automatic selection or ejection of expert members based on their performance can be used to maintain the expertise of the group. The relevant documents provided by the expert group are sorted in rank order. To check the effectiveness of the system, we have developed several effectiveness measures based on rank order. In this paper we validate our approach with simulations of user feedback and expert group reorganisation, and evaluate the results using the new effectiveness measures. Our preliminary work was published in conference proceedings (Kim and Kim, 2001; Kim and Chung, 2001). Proposed method Dynamic authority weights of experts We define a group of people with high authority and much expertise in a special field as an “expert group”. Figure 1 shows a framework for a search engine with our recommender system. A meta-search engine is used to collect good web documents from the conventional search engines (e.g. Yahoo, AltaVista, Excite and InfoSeek). The addresses of the documents cited in the search engines are stored in the document database. Also recorded for each web document are details of how many search engines in the meta-search engine referred to the document, and how many times online users had accessed the web document using the search engine. For every category there is a list of top-ranked documents rated by an expert group, which are sorted by score. Authoritative webpages are determined by human expert group members. The experts examine the content of candidate webpages that are highly referenced among web documents or have been accessed by many users. The method of employing an expert group is based on the idea that for a given decision task requiring expert knowledge, many experts may be better than one if their individual An opinionbased decision model 587
DIR 33,3 web 88 Web Crawler Monitor search Indexer meta-search engine DBQuery Category Ranking Figure 1 Search eng chitecture expert group anking engine judgments are properly combined. In our system, experts decide whether a web document should be classified as a recommended document for a given category a simple way to combine the experts individual judgements is majority voting iere and Tadepalli, 1997; Li and Jain, 1998), where each expert has a binary vote for each web document and the documents obtaining equal to or greater than half of the votes are classified into a top-ranked list. An alternative method is a weighted linear combination, where a weighted linear um of expert voting yields the collaborative net-effect ratings of documents. In this paper, we take the adaptive weighted linear combination method, where the individual contributions of members of the expert groups are weighted by their evaluation performance. All the experts evaluations are summed with weighted linear combinations. The expert rating results will dynamically change depending on each experts performance. Our approach to expert group decision-making is similar to the classifier committee concept of Li and Jain(1998)and Sebastiani (1999), except that their methods use classifiers based on various statistical or learning techniques instead of human interactions and decisions. This weighted measure is useful even when the number of experts is not fixed. How to choose experts and decide authority weights is an issue. Initially, experts ill be selected from among the users who have most frequently rated products or documents. a positive authority weight will be assigned to each expert member. The voting results of experts will determine a score over a given document. The score ranking will reflect the importance of the document. As time goes on, the authority weight will be changed depen users'feedback. An expert will receive a higher authority weight if his or her agrees with those of general users, and
judgments are properly combined. In our system, experts decide whether a web document should be classified as a recommended document for a given category. A simple way to combine the experts’ individual judgements is majority voting (Liere and Tadepalli, 1997; Li and Jain, 1998), where each expert has a binary vote for each web document and the documents obtaining equal to or greater than half of the votes are classified into a top-ranked list. An alternative method is a weighted linear combination, where a weighted linear sum of expert voting yields the collaborative net-effect ratings of documents. In this paper, we take the adaptive weighted linear combination method, where the individual contributions of members of the expert groups are weighted by their evaluation performance. All the experts’ evaluations are summed with weighted linear combinations. The expert rating results will dynamically change depending on each expert’s performance. Our approach to expert group decision-making is similar to the classifier committee concept of Li and Jain (1998) and Sebastiani (1999), except that their methods use classifiers based on various statistical or learning techniques instead of human interactions and decisions. This weighted measure is useful even when the number of experts is not fixed. How to choose experts and decide authority weights is an issue. Initially, experts will be selected from among the users who have most frequently rated products or documents. A positive authority weight will be assigned to each expert member. The voting results of experts will determine a score over a given document. The score ranking will reflect the importance or popularity of the document. As time goes on, the authority weight will be changed depending on users’ feedback. An expert will receive a higher authority weight if his or her opinion agrees with those of general users, and Figure 1. Search engine architecture OIR 33,3 588