Unified extractive-abstractive summarization: a hybrid approach utilizing BERT and transformer models for enhanced document summarization

Key contributions

The key contributions are as follows:

• An efficient intermediate representation for each sentence in the document is generated by extracting semantical features using an embedding technique.

• A hybrid approach using both extractive and abstractive summarization techniques for generating concise and informative summary is proposed.

• For extractive summarization, a contextually firm system is developed that learns the context of the document end-to-end to extract the informative sentences eliminating the redundancy.

An abstractive summarizer is developed that takes the informative sentences obtained from the extractive summarizer to generate coherent summary that are similar to the human generated summary. The rest of this article is organized as follows: “Related Works” provides a brief overview of existing extractive and abstractive summarization techniques. “Methodology” details the methodology of the proposed hybrid summarization technique. “Experiments and Results” outlines the experimental setup and presents the results. Finally, “Conclusion and Future Works” concludes the article and discusses avenues for future research.

Pre-trained language models

Recent advancements in natural language processing (NLP) have seen the development of several language models (Nallapati, Zhai & Zhou, 2017; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018; Dong et al., 2019), resulting in significant progress across various NLP tasks. These models have expanded the scope of embeddings by extracting semantic representations from extensive datasets using language modelling objectives. One prominent model in this domain is Bidirectional Encoder Representations from Transformers (BERT) (Radford et al., 2018), designed for masked language modelling (MLM) and next sentence prediction (NSP) tasks, trained on a dataset comprising 3,300 million words. BERT is sensitive to word order and incorporates symmetric regularization into its learning process.

The preprocessing of input text involves the insertion of specific tokens at the beginning and end of each sentence. The comprehensive information from the entire input sequence is captured through output representations, with the token at the beginning and end signifying the boundaries of a sentence. This processed text is represented as a sequence of tokens, with each token characterized by token embedding, segmentation embedding, and position embeddings. Token embedding captures the meaning of each token, segmentation embeddings distinguish between two sentences, and position embeddings denote the position of each token in the input sequence. These embeddings are aggregated into a single vector and passed through multiple layers in the bidirectional transformer.

The bidirectional transformer model generates contextual representations, utilizing multiple attention heads for each input sequence. This representation is expressed as:

(1) $${\tilde{e}}^l=LN\left(e+MAtt\left(e^{l-1}\right)\right)$$

(2) $$e^l=LN\left({\tilde{e}}^l+FFN\left({\tilde{e}}^l\right)\right)$$where, $e^0=x$ are the input vectors, LN is the normalization function for each layer, Multiple head attention operation is representaed as MATT, a double layer feed forward neural network operation is given as FFN. $l$ represents the depth of the stacked layer.

Output vector obtained from BERT comprises rich context-sensitive information. Such a pre-trained model serves to upgrade the effectiveness of language understanding tasks. Currently, pre-trained models are imposed in several generation challenges (Rothe, Narayan & Severyn, 2020; Zhang et al., 2018). While fine-tuning for a definite task, unlike other traditional models like ELMo (has fixed parameters), parameters are collectively fine-tuned with supplemental task-oriented parameters.

Extractive summarization

Summary generation is done by identifying the most significant sentences in the document using the extractive summarization technique. Neural network-based summarization model handles the extractive summarization task as a sentence classification problem. Neural encoder phase produces representations for each sentence and classifier identifies the sentences that have to be included in the summary. The earliest neural network approach for extractive summarization is SummaRuNNer (See, Liu & Manning, 2017) which enacts an RNN-based encoder. A reinforcement learning-based summarization model is Refresh (Narayan, Cohen & Lapata, 2018a) in which training is performed by global optimization of the ROUGE evaluation metric. Latent model (Zhang et al., 2018) projects extractive summarization as an inference problem with latent variables that directly optimizes the susceptibility of human summaries given selective summary sentences rather than just optimizing the susceptibility of assigned labels. In a model termed Sumo (Liu, Titov & Lapata, 2019) multi-root dependency tree representation of the source document is provoked with an idea of structured attention during the process of identifying the output summary. A state-of-the-art model in extractive summarization termed NeuSum (Zhou et al., 2018) concurrently scores and picks sentences for generating a summary.

Abstractive summarization

Neural network-based approaches address the abstractive summarization task as a problem that has to be handled one sequence after the other. Encoder phase corresponds to the series of tokens in the input document $t=\{t_1\text{,}{t}_2\text{,}\dots \text{,}{t}_n\}$to a serious of consecutive representations $z=\{z_1\text{,}{z}_2\text{,}\dots \text{,}{z}_n\}$, whereas the decoder phase, produces the summary $y=\{y_1\text{,}{y}_2\text{,}\dots \text{,}{y}_m\}$token-by-token in an auto-regressive fashion, therefore modelling the contingent probability $p\left(y_1\text{,}{y}_2\text{,}\dots \text{,}{y}_m|t_1\text{,}{t}_2\text{,}\dots \text{,}{t}_n\right)$.

Rush, Chopra & Weston (2015) and Nallapati et al. (2016) proposed a model that implemented the encoder-decoder architecture for summarization. An improvement is performed in this model, PTGEN (See, Liu & Manning, 2017) is proposed by applying a pointer generator network that admits the reproduction of words from the input text and a Cov (coverage) mechanism for tracing words towards summarization. An abstractive summarization model (See, Liu & Manning, 2017) is proposed where a hierarchical attention mechanism facilitates numerous factors to collectively represent the documents for decoding. End-to-end training is performed on their deep communication agent (DCA) using reinforcement learning. An abstractive summarization approach (Celikyilmaz et al., 2018) that incorporates Deep Reinforcement Model (DRM) utilizes intra-attention mechanism to solve the coverage problem. BOTTOMUP (Gehrmann, Deng & Rush, 2018) approach for summarization incorporates content selector that initially identifies the phrases from the source document to be included in the summary and decoder handles the preselected phrases alone with copy mechanism. An extremely abstractive summarization (single sentence summary) model (TConvS2S) (Narayan, Cohen & Lapata, 2018b) based on convolution neural network (CNN) is proposed with a constraint on topic distributions.

A hybrid extractive-abstractive summarization

In general, the human mind practices both extractive and abstractive approaches to summarize any document (Jing et al., 2000). Initially the significant parts in the document are identified and extracted. Further, based on the extracted content, decisions are made on what to ignore, paraphrase and transform them into precise and understandable summary. Such hybrid models can be split into three main categories (i) depending on attention for consistency; (ii) word-level evidence or statistical methods for generating summary; (iii) training the content generator using pseudo labels or other heuristics.

A hybrid summarization model (Hsu et al., 2018) that integrates the benefits of extractive and abstractive summarization is proposed. Sentence-level attention facilitates the regulation of word-level attention in a manner that tokens that are less attended in a sentence are less likely to be considered. In addition to this, an inconsistency loss function is used to indicate the inconsistency between two levels of attention.

Initial approaches involved PEGASUS model (Zhang et al., 2020) that was trained to summarize long-documents. As the model was much complicated and more topic specific, generalizability was compromised. Another challenge with this model is the inefficient context comprehension. To make the model better and cost-effective, most of the researchers have contributed to introduce new parameters which are derived weights from large-pre-trained models. To optimize the selection of the hyperparameters, Fast Fourier Transform (FFT) function is integrated to ignore redundancies in hidden sequence to accomplish discrete cosine transformation (DCT) (He et al., 2023).

This reduces the computational cost but failed to retain the benefits of various large pre-trained models like bidirectionality, transfer learning, etc., (Edunov, Baevski & Auli, 2019) The EASE model (Li et al., 2021) relies on the principle of information bottleneck (IB) that represents the understanding between the size of the extracted proof content and the knowledge provided for generating the final summary. Ravaut, Joty & Chen (2022) utilized a simple but effective extractive and abstractive hybrid model (Ravaut, Joty & Chen, 2022) where the first phase identifies the primary sentences and combines them to generate a summary.

Gap analysis

Most of the existing methods are task specific and involve numerous parameters for the conceptual summary generation but miss to consider the semantical relevance between the sentences of the input document. A document encompasses topic specific information which is more detailed and redundant. The redundant data must be removed and significant information has to be considered to generate an informative and concise summary. The key motive of our proposed model is to develop a contextually firm system that can learn the contexts from end-to-end. This is done by extracting contextual features from the sentences in the source document through which the informative sentences are identified. By taking those informative sentences as input, an abstractive summary is generated which is precise, informative and similar to the human generated summary. Although, the recent summarization techniques incorporate pre-trained models and large language models, it could not generate summary which is exactly similar to an expected summary. This affects the ROUGE score to a greater extent. For low resource languages, training cannot be performed effectively due to the unavailability of datasets with efficient reference summary. This research work relies on the semantic relevance of contextual features for the generation of summary and is compared with the recent summarization techniques that are focused on quality summary.

Problem description

Summarization aims in generating a summary that is readable, relevant, simple and informative without any redundancy. Given an input document D, consisting of multiple sentences $D=\left\{S_1\text{,}{S}_2\text{,}\dots \text{,}{S}_n\right\}$, where $S_i$ is the ith sentence in the document. Extractive summarization uses sentence selection $S_{e1}\text{,}{S}_{e2}\text{,}\dots \text{,}{S}_{en}\text{,}$ with an assumption that the selected summary sentences comprise significant and relevant information from the document. This is accomplished by depicting the relationship between sentences. $Z=\left\{Z_1\text{,}{Z}_2\text{,}\dots \text{,}{Z}_m\right\}$ represents ‘m’ the number of significant sentences extracted from source document. Extracted informative sentences (Z) are considered as input for generating an abstractive summary. Language generation model is incorporated to form new tokens and phrases to generate an abstractive summary (Y). Figure 1 illustrates the architecture of the proposed hybrid summarization model. The details of each phase are explained below.

Document pre-processing

In general, a document contains pictures, tables, references, symbolic representations, titles which will not contribute much to the semantic representation. Input document is broken down as a list of sentences which is termed as ‘Sentence Tokenization’. Few sentences in the document are semantically redundant and irrelevant, such elements can be ignored from the input document before being processed further. The position and size of the sentences plays a vital role in identifying the significance. Sentences which lie at the beginning of the document generally emphasize the knowledge about the document’s domain. Longer sentences hold more information whereas the shorter sentences have less information which means that the sentences that are long are given more priority. Pre-processing methods such as tokenization, ignoring short sentences, removal of symbols and other techniques are applied to the input document for analyzing and cleaning for further processing.

Sentence representation

Bidirectional Encoder Representation using Transformers (BERT) (Devlin et al., 2018) is a deep bidirectional model designed to generate representations from the unlabelled text. During the pre-processing phase, each input token is represented to the model using a specific set of rules. Each input embedding is a conjunction of position, segment and token embeddings. This deep bidirectional model acquires information from both directions during the phase of training. This model is pre-trained on two NLP tasks, masked language modelling (MLM) and next sentence prediction (NSP). This model is trained to predict masked/hidden words in the sentence. It is also trained to predict the following sentence of the given input sentence which deserves the knowledge of association between sentences. Being trained for these two tasks, this model holds the capacity to generate a conceptualized representation for each given input unit.

Sentence embedding is performed using Sentence Bidirectional Encoder Representation using Transformers (SBERT) (Radford et al., 2018) which is an alteration of BERT network that incorporates Siamese and triplet network to obtain contextually sensible vector representation of sentences. BERT involves a cross-encoder architecture and pair of sentences are fed as input to the transformer network to predict the target value which may not be suitable for several regression tasks because of possible associations. A general way to perform clustering is by associating each sentence to a vector space in such a way that semantically similar sentences lie nearer to each other. This led to the idea of picking an individual sentence as an input to BERT to obtain its embeddings. To improve the effectiveness of the embeddings in such an approach, SBERT is utilized. This network architecture empowers the computation of static length vectors for input sentences.

SBERT is fine-tuned on natural language inference (NLI) data to generate sentence embeddings that surpass other sentence embedding techniques. SBERT utilizes the pre-trained BERT and fine-tuning is done to produce efficient embeddings and minimize the training time. A pooling layer is appended on top of the BERT output to return static-sized embedding. A Siamese and triplet network (Schroff, Kalenichenko & Philbin, 2015) is built to upgrade the weights to create semantically meaningful embeddings. These embeddings facilitate semantic similarity computation using distance calculation metrics. The structure of the network relies on the type of training data (labelled or unlabeled) and is tested with the following objective functions.

Objective function for classification: The embeddings for an input sentence pair $\left(u,v\right)$is appended with the element-wise difference $\left|u,v\right|$ and is multiplied with trainable weights $W_t\in R^{3n\times k}$,

(3) $$o=softmax\left(W_t\left(u,v,\left|u-v\right|\right)\right)$$

Here n is the sentence embedding dimension and k is the label count. The cross-entropy loss is optimized.

Objective function for regression: The distance between two sentence embeddings is computed using a cosine similarity measure and Mean-squared-error loss is applied as the objective function.

Objective function for triplet network: Three sentences is considered for tuning the loss in the network. Initially, an anchor sentence (a) is picked. A positive sentence (p) having the same label as that of the anchor sentence is chosen and another sentence (n) having the other label is chosen. The triplet loss is estimated in a manner that the gap between ‘a’ and ‘p’ is less than the gap between ‘a’ and ‘n’. This estimation is utilized to tune the weights associated with the network. Mathematically,

(4) $$max\left(\left|\left|S_a-S_p\left||-|\right|S_a-S_n\right|\right|+ϵ,0\right).$$

Here, $S_a,S_{p}and{\mathrm{S}}_{\mathrm{n}}$ represents the sentence embeddings of sentences a, p and n respectively. $\left||.|\right|$ represents the distance measure and ϵ is the margin that projects that $S_{p}isatleastϵcloserto{S}_{p}than{S}_n.$ In general Euclidean distance is applied to compute the distance with the margin ϵ set as 1 (ϵ = 1).

SBERT for regression generates the representations and performs distance estimation using the cosine similarity calculation method. The semantical representations generated by the BERT model analyzes the input on both token level and sentence level and so is capable of handling diversity on all aspects. In the proposed model the conceptual representation for the pair of pre-processed sentences is fetched before distance estimation and is considered as input for further processing. Steps incorporated in the generation of intermediate representation for each sentence are illustrated in Fig. 2.

Proposed hybrid summarization model

In general, a document comprises a greater number of sentences which makes the summary generation complex. To deal with this challenge, the proposed model integrates extractive and abstractive summarization techniques (Algorithm 1). To facilitate effective summarization, redundant and irrelevant information has to be ignored for the selection of significant sentences. A novel extractive summarization model is proposed to identify significant sentences from the source document. These extracted significant sentences are taken as input for generating an abstractive summary, that is similar to the manually created summary.

Significant sentence extraction

A novel extractive summarization system takes the representation of each sentence in the input document as input and identifies the most informative sentences. The semantic relevancy between sentences is identified by estimating the distance between each sentence representations. Several distance calculation metrics (Huang, 2008) are available to estimate the relevancy between sentences. Manhattan distance estimation is proven to be effective in determining the distance between high-dimensional data (Aggarwal, Hinneburg & Keim, 2001). This works by calculating the covered distance while tracking a grid-like path between two points. Considering two high-dimensional points $x=x_1\text{,}{x}_2\dots x_i$ and $y=y_1\text{,}{y}_2\dots y_i,$the Manhattan distance measurement is performed using,

(5) $$d=\sum _{i=1}^n\left|x_i-y_i\right|$$where, n is the number of dimensions, $x_i$ and $y_i$ are high-dimensional data points (sentence representations) and d is the distance between two sentence representations. A greater distance value between two sentences indicates that the sentences are dissimilar. An adjacency matrix is generated with rows representing the sentence ids and columns representing the elements holding the distance value between two corresponding sentences. This matrix has 0’s in its diagonal value as the distance between a sentence with itself is zero. From this adjacency matrix, a rank matrix is derived that ranks each sentence based on its distance from other sentences. In a rank matrix, the rows represent the sentence ids and the columns represent the rank. While constructing the rank matrix, the diagonal 0’s is eliminated. During the generation of the rank matrix, the elements of each row in the distance matrix are sorted in ascending order and the values are replaced with their respective sentence ids. Each column in the rank matrix represents the level of closeness among sentences. Concerning the first column, the most frequent sentence id is considered the primary sentence which is highly relevant to all the other sentences. The row of the selected primary sentence id are taken as the summary sentences. The selection of summary sentences is based on the desired summary size. Finally, the summary sentences are arranged sequentially according to the given input document.

Consider an input passage having six sentences $S=\left\{S_1\text{,}{S}_2\text{,}{S}_3\text{,}{S}_4\text{,}{S}_5\text{,}{S}_6\right\}$, whose representations are generated using Sentence BERT. The distance matrix is generated by computing the distance between sentences. The distance matrix having zeros in its diagonal shows that the distance between a sentence with itself is zero. From this distance matrix, a rank matrix is generated which ranks the sentences based on its similarity. In the rank matrix, the row represents the sentence ids and the elements in the row are arranged based on the closeness with the other sentences. In the above example from Fig. 3, for sentence ID 1, the elements are arranged as S2, S4, S3, S6 and S5. The S1 is not considered as the closeness to itself is 0. This reduces one column from the distance matrix while constructing the rank matrix. The size of the rank matrix is $\left(numberofsentences\mathrm{x}numberofrank\right)$ and the number of ranks estimated is $\left(Numberofinputsentenes-1\right).$ The first column holds the sentence that is more close to each other sentences. The sentence ids that are most frequent in the first column are taken as the primary sentence, which is closer to all the other sentences. In situations where two different sentence ids occur the same number of times, track the second column and identify the sentence id that is most frequent. Each column in the rank matrix shows the level of closeness between sentences. Elements in selected row are sorted in ascending order and the distance values are replaced with their corresponding sentence ids. In the given example, the first column in the rank matrix $S_1$ is most frequent, which means that, the sentence $S_1$ is semantically similar to all the other sentences. Thus, $S_1$ is considered as the primary sentence. The row of the selected sentence id indicates the summary sentences. Here, the elements in the row $S_1$ is $\left\{S_2\text{,}{S}_4\text{,}{S}_3\text{,}{S}_6\text{,}{S}_5\right\}$, which states the hierarchy in which these sentences are closer to $S_1.$ Selection of summary sentences from this set $\left\{S_1\text{,}{S}_2\text{,}{S}_4\text{,}{S}_3\text{,}{S}_6\text{,}{S}_5\right\}$, is based on the desired summary size. Finally, the summary sentences are arranged sequentially according to the given input document. These extracted summary sentences are considered as input for generating an abstractive summary. The sentences extracted from the input document based on the desired summary size are presented in Table 1.

Table 1:

Significant sentence selection outputs for the above-given sample.

Input document sentences	$\left\{{\mathit{S}}_1\text{,}{\mathit{S}}_2\text{,}{\mathit{S}}_3\text{,}{\mathit{S}}_4\text{,}{\mathit{S}}_5\text{,}{\mathit{S}}_6\right\}$
Primary sentence ID	$\left\{S_1\right\}$
Sentences closer to primary sentence	$\left\{S_2\text{,}{S}_4\text{,}{S}_3\text{,}{S}_6\text{,}{S}_5\right\}$
Summary sentences along with primary sentence.	$\left\{S_1\text{,}{S}_2\text{,}{S}_4\text{,}{S}_3\text{,}{S}_6\text{,}{S}_5\right\}$
25% of desired summary	$\left\{S_1\text{,}{S}_2\text{,}\right\}$
50% of desired summary	$\left\{S_1\text{,}{S}_2\text{,}{S}_4\right\}$
80% of desired summary	$\left\{S_1\text{,}{S}_2\text{,}{S}_3\text{,}{S}_4\text{,}{S}_6\right\}$

DOI: 10.7717/peerj-cs.2424/table-1

Language generation

Followed by the extraction of a certain number of significant and semantically relevant sentences from the input document, a model that learns to originate an abstractive summary is incorporated. Here the significant sentences extracted are denoted as X $=\left\{X_1\text{,}{X}_2\text{,}{X}_3\text{,}{X}_4\text{,}{X}_5\text{,}{X}_6\right\}$, where $X_iϵn$ represents the representations of the extracted sentences for which the abstractive summary has to be created. Two main challenges to be addressed during abstractive summarization are: (1) complete context has to be considered by the decoder during the prediction of words. (2) Exploiting pre-trained contextualized languages on the decoder phase to facilitate learning of summary representations, contextual interactions and modelling of languages collaboratively. To generate an abstractive summary, BERT based language generation model is incorporated (Zhang, Xu & Wang, 2019). This model utilizes a pre-trained language model during its encoder and decoder process and the training is performed end-to-end without manually extracted features. This model generates each word in the summary by exploring the contextual information from both sides. The functioning of language generation using BERT based model is illustrated in Fig. 4. Summary draft generation is rooted in the sequence-to-sequence network. Extracted significant sentences are considered as the input document for generating an abstractive summary. The model used for generating language has two phases, encoder and decoder. The encoder phase encodes the input document X $=\left\{X_1\text{,}{X}_2\text{,}\dots \text{,}{X}_m\right\}$, into intermediate representation R $=\left\{R_1\text{,}{R}_2\text{,}{R}_3\text{,}\dots \text{,}{R}_m\right\}$. BERT model is used as the encoder which initially generates word embeddings for the input sequence and then computes document embeddings, which is the output of the encoder.

(6) $$R=BERT\left(X\right).$$

This representation R is given as input to the decoder for the generation of summary draft D $=\left\{D_1\text{,}{D}_2\text{,}\dots \text{,}{D}_{\left|d\right|}\right\}.$The summary draft decoder comprises of word embeddings matrix generated by BERT to associate the output of the precedent summary draft into generated embedding vectors at every time step. As the entire input sequence is not fed as input to the decoder, the BERT model is not employed for the prediction of context vectors. An N-layer Transformer decoder is implemented to obtain the conditional probability $P\left(D|R\right)$. The multi-head attention mechanism in the Transformer’s encoder-decoder facilitates the decoder to acquire smooth alignment between input document and summary. At each timestep, the prediction of output probability conditioned on the encoder’s representation and precedent output is estimated by the summary draft decoder. Each sequence generated is truncated at the occurrence of the ‘[PAD]’ token. The summary draft decoder integrates the output from the multihead attention decoder and the actual summary draft output. The learning objective of the decoder is to alleviate the negative likeliness of conditional probability. step

The bi-linear dot product of the final layer decoder output of the Transformer and the encoder output is initially estimated to compute the attention probability distribution over input document X. The choice of words in the summary is performed using a copy mechanism (Gu et al., 2016) based on the Transformer decoder to avoid including out-of-vocabulary tokens. Calculation of copying gate

(7) $$g_tϵ\left[0,1\right]$$that brings a smooth preference between choosing a word from an input document and building from the vocabulary. The weighted sum of copy probability and generation probability is computed to obtain the final probability of extended vocabulary $V+X$, where $X$ is the out-of-vocabulary words from input document.

(8) $$g_t=sigmoid\left(w_g\left[e_t,s\right]+b\right)$$where, $W_g$ is a linear transformation with g indicating the token significance, $e_t$ is the output from the decoder’s last layer, s is the output from the encoder, b is the bias and ${\hat{e}}_t$ manages the significant feature information is calculated as

(9) $${\hat{e}}_t=g_t{e}_t$$

The refining process is to improve the efficiency of the decoder by applying BERT’s contextualized representations to ignore modifying the encoder for re-utilization. A word-level refined decoder is included in the decoder phase that takes generated summary draft as input and the refined summary is generated as output. This initially masks every word in the draft one after the other and is given as input to the BERT model for the generation of context vectors. A refined summary word is predicted by applying an N-layer Transformer decoder, similar to the draft decoder. At each time step, a specific word in the summary draft is masked, from which the refined word is predicted by taking other words as the input. Thus, the refined decoding stage generates a complete and consistent sequence with their pre-training processes. Thus, the language generation process works as follows: Initially, a summary draft is generated by the draft decoder based on the input document (extracted significant sentences). This summary draft is modified using the refine decoder. Each word in the summary draft is focused on refinement, as the execution is similar to the BERT pre-training process. With the capability of the contextual language model, the decoder can generate more fluent and complete sequences.

Experimental data

Implementation of the proposed hybrid summarization model is performed using packages such as spacy, pandas, NumPy, sklearn and Transformers. The proposed model is applied to a benchmark dataset and its performance is assessed with ROUGE metrics implemented using PyRouge package. The details of the dataset and the performance metrics used to evaluate the generated summary are discussed below.

Dataset

The performance of the proposed model is examined on English language-based news dataset CNN/DailyMail. This dataset holds a huge volume of news articles developed by CNN and DailyMail news reporters. A non-anonymized version of this dataset (See, Liu & Manning, 2017) is purely designed for machine-reading understanding and question answering. The latter version of this dataset (Hermann et al., 2015) is compatible with extractive and abstractive summarization tasks. This constitutes textual news passages and the respective summary with the highlights. This includes 280,000 training samples, 13,368 verification samples and 11,490 test set samples approximately. Each news passage contains 40 sentences and 781 tokens on average. The news passages from this dataset are taken as input to the proposed hybrid summarization model and the summary is generated. The generated summary is then compared with the reference summary to assess the summary quality, which contributes to the evaluation of the hybrid summarization model.

Evaluation metrics

The summary quality is evaluated using ROUGE metrics (Lin, 2004), an extensively used quantified metric. ROUGE score is determined through the overlaying words among the generated and reference summaries.

(10) $$ROUG{E}_n={\displaystyle \frac{\sum sϵ\left[Referencesummaries\right]\sum gra{m}_{nϵS}Coun{t}_{match}\left(gra{m}_n\right)}{\sum sϵ\left[Referencesummaries\right]\sum gra{m}_{nϵS}Count\left(gra{m}_n\right)}.}$$

Several variants of ROUGE such as ROUGE-1, ROUGE-2 and ROUGE-L are generally utilized for evaluation. Summary information is assessed using ROUGE-1 and ROUGE-2 and summary fluency is evaluated using ROUGE-L (Longest common subsequence). Assume the reference summary as R, where $R=\left\{r_1\text{,}{r}_2\text{,}{r}_3\text{,}\dots \text{,}{r}_i\right\}$ and $r_i$ is the ith word in the summary. System generated summary is represented as S, where $S=\left\{s_1\text{,}{s}_2\text{,}{s}_3\text{,}\dots \text{,}{s}_j\right\}$ and $s_j$ is the jth word in the summary. The estimation of precision, recall and F1-score of ROUGE-N is determined as follows.

(11) $$Precision={\displaystyle \frac{n\left(R\cap S\right)}{n\left(S\right)}}$$

(12) $$Recall={\displaystyle \frac{n\left(R\cap S\right)}{n\left(R\right)}}$$

(13) $$F1-Score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}.}$$

Here, n represents the n-gram unit and n(X) shows the size of set X. If n = 1, the overlaying of unigram words is estimated. Overlaying of the longest matching sequence of words is estimated using ROUGE-L. F1-score of various ROUGE metrics is utilized for evaluating the system-generated summary.

Experimental results