Automatic Sentence Ordering Assessment Based on Similarity

Abstract

One of the tasks of text generation is sentence ordering since it is crucial for readability. Nevertheless, there is no common approach for evaluation of sentence ordering. The state-of-the art methods are based on the comparison with a human-provided order. However, in many cases it is impossible or time and resource consuming. Therefore, we propose three completely automatic approaches for sentence order assessment where the similarity between adjacent sentences is used as a measure of text coherence. We showed that the methods based on word and noun similarities have very high agreement with the human-provided judgment. We also propose an automatic evaluation framework for analysis of the metrics of sentence order that requires only a text collection.