OATAO - Open Archive Toulouse Archive Ouverte Open Access Week

Some thoughts from IRIT about the scenario A of the TREC RTS 2016 and 2017 tracks

Hubert, Gilles and Moreno, José G. and Pinel-Sauvagnat, Karen and Pitarch, Yoann Some thoughts from IRIT about the scenario A of the TREC RTS 2016 and 2017 tracks. (2017) In: 26th Text REtrieval Conference (TREC 2017), 15 November 2017 - 17 November 2017 (Gaithersburg, Maryland, United States).

(Document in English)

PDF (Author's version) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: https://trec.nist.gov/pubs/trec26/papers/IRIT2-RT.pdf


The TREC Real-Time Summarization (RTS) track provides a framework for evaluating systems monitoring the Twitter stream and pushing tweets to users according to given profiles. It includes metrics, files, settings and hypothesis provided by the organizers. In this work, we perform a thorough analysis of each component of the framework used for batch evaluation of scenario A in 2016 and 2017. We found some weaknesses of the metrics and took advantage of these limitations to submit our official run in the 2017 edition. The good evaluation results validate our findings. This paper also gives clear recommendations to fairly reuse the collection.

Item Type:Conference or Workshop Item (Paper)
Additional Information:NIST: National Institute of Standards and Technology
HAL Id:hal-02319737
Audience (conference):International conference proceedings
Uncontrolled Keywords:
Institution:Université de Toulouse > Institut National Polytechnique de Toulouse - Toulouse INP (FRANCE)
French research institutions > Centre National de la Recherche Scientifique - CNRS (FRANCE)
Université de Toulouse > Université Toulouse III - Paul Sabatier - UT3 (FRANCE)
Université de Toulouse > Université Toulouse - Jean Jaurès - UT2J (FRANCE)
Université de Toulouse > Université Toulouse 1 Capitole - UT1 (FRANCE)
Laboratory name:
Deposited On:04 Oct 2019 13:50

Repository Staff Only: item control page