Evaluating Search Engines in chapter 8 of the book Search Engines Information Retrieval in Practice http://net. Hongfei Yan School of EECS, peking University 3/28/2011 Refer to the book s slides
Evaluating Search Engines in chapter 8 of the book Search Engines Information Retrieval in Practice http://net.pku.edu.cn/~course/cs410/2011/ Hongfei Yan School of EECS, Peking University 3/28/2011 Refer to the book’s slides
08: Evaluating Search Engines 8. 1 Why Evaluate 8.2 The Evaluation Corpus 8.3 Logging + 8.4 Effectiveness Metrics(+) 8.5 Efficiency Metrics 8.6 Training, Testing, and statistics +) 8.7 The bottom Line skip) 3/N
08: Evaluating Search Engines 8.1 Why Evaluate 8.2 The Evaluation Corpus 8.3 Logging (+) 8.4 Effectiveness Metrics (+) 8.5 Efficiency Metrics 8.6 Training, Testing, and Statistics (+) 8.7 The Bottom Line (skip) 3/N
Search engine design and the core information retrieval issues Relevance Performance Effective ranking -Efficient search and indexing Evaluation Incorporating new data -Testing and measuring Coverage and freshness Information needs Scalability User interaction Growing with data and users Adaptability Tuning for applications Specific problems E.g…spam 4/N
Search engine design and the core information retrieval issues Relevance -Effective ranking Evaluation -Testing and measuring Information needs -User interaction Performance -Efficient search and indexing Incorporating new data -Coverage and freshness Scalability -Growing with data and users Adaptability -Tuning for applications Specific problems -E.g., spam 4/N
Evaluation Evaluation is key to building effective and efficient search engines measurement usually carried out in controlled laboratory experiments online testing can also be done Effectiveness, efficiency and cost are related e. g if we want a particular level of effectiveness and efficiency this will determine the cost of the system configuration efficiency and cost targets may impact effectiveness 5/N
Evaluation • Evaluation is key to building effective and efficientsearch engines – measurement usually carried out in controlled laboratory experiments – online testing can also be done • Effectiveness, efficiency and cost are related – e.g., if we want a particular level of effectiveness and efficiency, this will determine the cost of the system configuration – efficiency and cost targets may impact effectiveness 5/N
08: Evaluating Search Engines 8. 1 Why Evaluate 8.2 The Evaluation Corpus 8.3 Logging 8. 4 Effectiveness metrics 8.5 Efficiency Metrics 8.6 Training, Testing, and Statistics 6/N
08: Evaluating Search Engines 8.1 Why Evaluate 8.2 The Evaluation Corpus 8.3 Logging 8.4 Effectiveness Metrics 8.5 Efficiency Metrics 8.6 Training, Testing, and Statistics 6/N