Evaluation Measures for Text Summarization

Year, pages: 2009, 251 - 275

We explain the ideas of automatic text summarization approaches and the taxonomy of summary evaluation methods. Moreover, we propose a new evaluation measure for assessing the quality of a summary. The core of the measure is covered by Latent Semantic Analysis (LSA) which can capture the main topics of a document. The summarization systems are ranked according to the similarity of the main topics of their summaries and their reference documents. Results show a high correlation between human rankings and the LSA-based evaluation measure. The measure is designed to compare a summary with its full text. It can compare a summary with a human written abstract as well; however, in this case using a standard ROUGE measure gives more precise results. Nevertheless, if abstracts are not available for a given corpus, using the LSA-based measure is an appropriate choice.

Steinberger, J., Ježek, K. 2009. Evaluation Measures for Text Summarization. In Computing and Informatics, vol. 28, no.2, pp. 251-275. 1335-9150.

Steinberger, J., Ježek, K. (2009). Evaluation Measures for Text Summarization. Computing and Informatics, 28(2), 251-275. 1335-9150.