OATAO - Open Archive Toulouse Archive Ouverte Open Access Week

Human-Based Query Difficulty Prediction

Chifu, Adrian-Gabriel and Déjean, Sébastien and Mizzaro, Stefano and Mothe, Josiane Human-Based Query Difficulty Prediction. (2017) In: 39th European Colloquium on Information Retrieval (ECIR 2017), 9 April 2017 - 13 April 2017 (Aberdeen, Scotland, United Kingdom).

(Document in English)

PDF (Author's version) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: https://doi.org/10.1007/978-3-319-56608-5_27


The purpose of an automatic query difficulty predictor is to decide whether an information retrieval system is able to provide the most appropriate answer for a current query. Researchers have investigated many types of automatic query difficulty predictors. These are mostly related to how search engines process queries and documents: they are based on the inner workings of searching/ranking system functions, and therefore they do not provide any really insightful explanation as to the reasons for the difficulty, and they neglect user-oriented aspects. In this paper we study if humans can provide useful explanations, or reasons, of why they think a query will be easy or difficult for a search engine.We run two experiments with variations in the TREC reference collection, the amount of information available about the query, and the method of annotation generation. We examine the correlation between the human prediction, the reasons they provide, the automatic prediction, and the actual system effectiveness. The main findings of this study are twofold. First, we confirm the result of previous studies stating that human predictions correlate only weakly with system effectiveness. Second, and probably more important, after analyzing the reasons given by the annotators we find that: (i) overall, the reasons seem coherent, sensible, and informative; (ii) humans have an accurate picture of some query or term characteristics; and (iii) yet, they cannot reliably predict system/query difficulty.

Item Type:Conference or Workshop Item (Paper)
Additional Information:Thanks to Springer editor. This papers appears in Volume 10193 of Lecture Notes in Computer Science ISSN : 0302-9743 ISBN: 978-3-319-56607-8 The original PDF is available at: https://link.springer.com/chapter/10.1007/978-3-319-56608-5_27
HAL Id:hal-01712541
Audience (conference):International conference proceedings
Uncontrolled Keywords:
Institution:Other partners > Aix-Marseille Université - AMU (FRANCE)
French research institutions > Centre National de la Recherche Scientifique - CNRS (FRANCE)
Université de Toulouse > Institut National Polytechnique de Toulouse - Toulouse INP (FRANCE)
Université de Toulouse > Université Toulouse III - Paul Sabatier - UT3 (FRANCE)
Université de Toulouse > Université Toulouse - Jean Jaurès - UT2J (FRANCE)
Université de Toulouse > Université Toulouse 1 Capitole - UT1 (FRANCE)
Other partners > Università degli studi di Udine - UNIUD (ITALY)
Laboratory name:
Deposited On:01 Feb 2018 13:22

Repository Staff Only: item control page