SWIP at QALD-3 : results, criticisms and lesson learned

Abstract

This paper presents the results obtained by the SWIP system while participating in the QALD-3 (Question Answering over Linked Data) challenge, co-located with CLEF 2013 (Conference and Labs of the Evaluation Forum). We tackled task 1, multilingual question answering, whose purpose is to interpret natural language questions in order to return the answers contained in a graph knowledge base. We answered queries of both proposed datasets (one concerning DBpedia, the other Musicbrainz) and took into consideration only questions in English. The system SWIP (Semantic Web Interface using Patterns) aims at automatically generating formal queries from user queries expressed in natural language. For this, it relies on the use of query patterns which enable the complex task of interpreting natural language queries. The results obtained on the Musicbrainz dataset (precision = 0,51, recall = 0,51, F-measure = 0,51) are very satisfactory and encouraging. The results on DBpedia (precision = 0,16, recall = 0,15, F-measure = 0,16) are more disappointing. In this paper, we present both the SWIP approach and its implementation. We then present the results of the challenge in more detail and their analysis. Finally we draw some conclusions on the strengths and weaknesses of our approach, and suggest ways to improve its performance.