Keynote Talk:
Craig Macdonald
School of Computing Science, University of Glasgow, UK
Bio
Craig Macdonald is Lecturer at the University of Glasgow, UK. Currently, his main research topics deal with Information Retrieval (IR) in general, for instance in settings such as Web, Enterprise, social media and Smart cities. He regularly participates in TREC, and jointly co-ordinated the TREC Blog track from 2006-2010, the Microblog track (from 2011-2012), and the Web track (2014-). He is a lead developer for the Terrier IR platform, and also uses Terrier in his research publications.
Hypothesis Testing for Risk-Sensitive Evaluation and Learning to Rank in Web Search
When a user is unsatisfied with the quality of results of a web search engine, they may switch to another, leading to a loss of ad revenue to the engine. Use of a robust retrieval approach is therefore essential, to that the experience of the users of the search engine is not damaged by poorly-performing queries. For this reason, there has been growing interest in measuring robustness using a new class of risk-sensitive evaluation measures, which assess the extent to which a system exhibit risk, i.e. performs worse than a given baseline system on a set of queries.
In this talk, we describe our recent advances in two families of risk-sensitive evaluation measures both based upon hypothesis testing, and their integration into a state-of-the-art learning to rank algorithm, to create effective yet robust retrieval models.
Firstly, we argue that risk-sensitive evaluation is akin to the underlying methodology of the Student's t-test for matched pairs. Hence, we introduce a risk-reward tradeoff measure TRisk that generalises the existing URisk measure, and which is theoretically grounded in statistical hypothesis testing.
Secondly, we argue that using a single system as the baseline suffers from the fact that retrieval performance highly varies among IR systems across topics. Thus, a single system would in general fail in providing enough information about the real baseline performance for every topic under consideration, and hence cannot in general measure the real risk associated with any given system. Based upon the Chi-squared statistic, we describe a second family of risk-reward tradeoff measures that take into account multiple baselines when measuring risk.
Experiments using 10,000 topics from the MSLR learning to rank dataset from the Bing search engine demonstrate that our proposed t-test and Chi-square based objective functions that reduces the number of poorly performing queries exhibited by a state-of-the-art learning to rank algorithm.
Djoerd Hiemstra
Department of Computer Science, University of Twente, The Netherlands
Bio
Djoerd Hiemstra is part-time associate professor at the University of Twente. He also heads Searsia, a University of Twente spin-off that develops an open source federated search engine. Djoerd contributed to over 200 research papers in the field of information retrieval, covering topics such as language models, structured information retrieval, multimedia retrieval, and federated web search. Djoerd published papers with research labs of Microsoft (where he did an internship in 2000), Yahoo (where he was a visiting researcher in 2008), and Yandex (which he visited in 2011).
The best learning-to-rank method of all: A final verdict?
Like most information retrieval methods, learning-to-rank methods are evaluated on benchmark datasets, such as the many datasets provided by Microsoft and the datasets provided by Yahoo and Yandex. Many of the learning-to-rank datasets offer feature set representations of the to-be-ranked documents instead of the documents themselves. Therefore, any difference in ranking performance is due to the ranking algorithm and not the features used. This opens up a unique opportunity for cross-benchmark comparison of learning-to-rank methods.
In this talk, I propose a way to compare learning to rank methods based on a sparse set of evaluation results on many benchmark datasets. Our comparison methodology consists of two components: (1) the Normalized Winning Number, a measure that gives insight in the ranking accuracy of the learning to rank method, and (2) the Ideal Winning Number, which gives insight in the degree of certainty concerning the ranking accuracy.
Evaluation results of 87 learning-to-rank methods on 20 well-known benchmark datasets are collected. I report on the best performing methods by Normalized Winning Number and Ideal Winner Number and suggest what methods need more research to make our analysis more robust.
Finally, we test the robustness of our results by comparing the results to situations where one of the datasets is not included in the analysis.
Workshop Schedule
- 14:00-14:10 - Opening and Welcome
- 14:10-15:00 - Keynote with Prof. Craig Macdonald, University of Glasgow
- 15:00-15:30 - Paper Session
-
Online Learning of a Ranking Formula for Revenue and Advertiser ROI Optimization
Or Levi
-
Query-Level Ranker Specialization
Rolf Jagerman, Harrie Oosterhuis and Maarten de Rijke.
- 15:30-16:00 - Coffee Break
- 16:00-16:50 - Keynote with Prof. Djoerd Hiemstra, University of Twente
- 16:50-17:20 - Paper Session
-
Discovery and Promotion of Subtopic Level High Quality Domains for Programming Queries in Web Search
Arpita Das, Saurabh Shrivastava and Manoj Chinnakotla
-
A Software Library for Conducting Large Scale Experiments on Learning to Rank Algorithms
Nicola Ferro, Paolo Picello and Gianmaria Silvello
-
The Impact of Negative Samples on Learning to Rank
Claudio Lucchese, Franco Maria Nardini, Raffaele Perego, and Salvatore Trani
- 17:20-17:50 - Short Invited Presentations
-
Multileaving for Online Evaluation of Rankers
Brian Brost
-
When Learning to Rank Meets Axiomatic Thinking
Hui Fang and Chengxiang Zhai
-
Learning to Rank Target Types for Entity-Bearing Queries
Darío Garigliotti and Krisztian Balog
- 17:50-18:00 - Wrap Up, Discussion and Closing
- 18:00 - Drinks at CASA 400