OATAO - Open Archive Toulouse Archive Ouverte Open Access Week

How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT

Morey, Mathieu and Muller, Philippe and Asher, Nicholas How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT. (2017) In: Conference on Empirical Methods on Natural Language Processing (EMNLP 2017), 7 September 2017 - 11 September 2017 (Copenhagen, Denmark).

[img]
Preview
(Document in English)

PDF (Author's version) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
146kB

Official URL: http://aclweb.org/anthology/D17-1137

Abstract

This article evaluates purported progress over the past years in RST discourse parsing. Several studies report a relative error reduction of 24 to 51% on all metrics that authors attribute to the introduction of distributed representations of discourse units. We replicate the standard evaluation of 9 parsers, 5 of which use distributed representations, from 8 studies published between 2013 and 2017, using their predictions on the test set of the RST-DT. Our main finding is that most recently reported increases in RST discourse parser performance are an artefact of differences in implementations of the evaluation procedure. We evaluate all these parsers with the standard Parseval procedure to provide a more accurate picture of the actual RST discourse parsers performance in standard evaluation settings. Under this more stringent procedure, the gains attributable to distributed representations represent at most a 16% relative error reduction on fully-labelled structures.

Item Type:Conference or Workshop Item (Paper)
Additional Information:The definitive version is available at : http://aclweb.org/anthology/D17-1137
HAL Id:hal-01650251
Audience (conference):International conference proceedings
Uncontrolled Keywords:
Institution:Université de Toulouse > Institut National Polytechnique de Toulouse - INPT (FRANCE)
French research institutions > Centre National de la Recherche Scientifique - CNRS (FRANCE)
Université de Toulouse > Université Toulouse III - Paul Sabatier - UPS (FRANCE)
Université de Toulouse > Université Toulouse - Jean Jaurès - UT2J (FRANCE)
Université de Toulouse > Université Toulouse 1 Capitole - UT1 (FRANCE)
Laboratory name:
Statistics:download
Deposited By: IRIT IRIT
Deposited On:24 Nov 2017 14:22

Repository Staff Only: item control page