Deep Learning Based Multi-document Summarization

Abstract

In this era of rapidly advancing technology, the exponential increase of data availability makes analyzing and understanding text files a tedious, labor-intensive, and time-consuming task. Multi-document summarization (MDS) is an effective tool for information aggregation that generates an informative and concise summary from a cluster of topic-related documents. In this thesis, we systematically over-viewed the recent deep learning based MDS models, proposed a series of novel methods to cope with the MDS tasks with deep learning technique and examine the behaviours of Transformer-based MDS models. Firstly, we presented a categorization scheme to organize current research and provide a comprehensive review for deep learning based MDS techniques, including deep learning based models, objective functions, benchmark datasets, and evaluation metrics. We reviewed development movements and provide a systematic overview and summary of the state-of-the-art. We also summarized nine network design strategies based on our extensive studies of the current models. Secondly, due to linguistic knowledge plays an important role in assisting models to learn informative representations, in this thesis, we presented a Transformer-based abstractive MDS method with linguistic-guided attention (LGA) mechanism for better representation learning. The proposed linguistic-guided attention mechanism can be seamlessly incorporated into multiple mainstream Transformer based summarization models to improve the quality of the generated summaries. We developed the proposed method based on Flat Transformer (FT) and Hierarchical Transformer (HT), named ParsingSum-FT and ParsingSum-HT respectively. Based on this work, we further proposed document-aware positional encoding and linguistic-guided encoding that can be fused with Transformer architecture for MDS. For documentaware positional encoding, we introduced a general protocol to guide the selection of document encoding functions. For linguistic-guided encoding, we presented to embed syntactic dependency relations into the dependency relation mask with a simple but effective non-linear encoding learner for feature learning. Empirical studies on both models demonstrate these two simple but effective methods can help the models outperform existing Transformer-based methods on the benchmark dataset by a large margin. Thirdly, the existing MDS methods neglect the specific information for each document, limiting the comprehensiveness of the generated summaries. To solve this problem, we presented to disentangle the specific content from documents in one document set. The document-specific representations, which are encouraged to be distant from each other via a proposed orthogonal constraint, are learned by the specific representation learner. We provided extensive analysis and had interesting findings that specific information can well-complementary with document set features for MDS tasks. Also, we found that the common (i.e. shared) information could not contribute much to the overall performance under the MDS settings. Fourthly, the utilization of Transformer based models prospers the growth of MDS. In order to thoroughly examine the behaviours of Transformer based MDS models, this thesis also presented five empirical studies on (1) measuring the impact of document separators quantitatively; (2) exploring the effectiveness of different mainstream Transformer structures; (3) examining the sensitivity of encoder and decoder (4) discussing different training strategies; (5) discovering the repetition in summary generation. The experimental results on two MDS datasets and eleven evaluation metrics show the influence of document separators, the granularity of different level features and different model training strategies. The experiments also indicated that the decoder exhibits greater sensitivity to noises in summarization tasks compared to the encoder, which indicates the important role played by the decoder, pointing a potential direction for future MDS researches. Furthermore, the experimental results indicated that the repetition problem in the generated summaries have correlations with the high uncertainty score. Finally, we discussed the open issues of deep learning based MDS and identified the future research directions of this field. We also proposed potential solutions for some discussed research directions.