Abstractive text summarization of low-resourced languages using deep learning

Related work

In recent years, Urdu linguistics has achieved significant progress. A substantial volume of data is generated by numerous portals and news websites day after day. Without knowing the meaning of the phrases, extractive summarization methods construct summaries (Dwi Sanyoto, 2017). As a result, abstractive summaries are more precise than extractive summaries (Kiyani & Tas, 2017). However, because statistical approaches are faster than linguistics procedures, the extracted summary is generated faster. Abstractive and extractive approaches for patent labelling have been examined (Moratanch & Chitrakala, 2016). Overall, comparing abstractive versus extractive (Dalal & Malik, 2013) approaches is difficult for various reasons. The approaches to text summarization are shown in Fig. 1. These are divided into extractive and abstractive text summarization based on the output type. The overview of extractive summarization types is depicted in Fig. 2 and the types of the overview of abstractive summarization are shown in Fig. 3.

1. Unsupervised learning: These methods do not require human summaries (user input) to determine the crucial aspects of the content.

   1. Graph-based approach: Since graphs may effectively reflect the document structure (Iyer, Chanussot & Bertozzi, 2018), these models are frequently employed in document summarization.

   2. Concept-based approach: This approach extracts (Hashemi, Tyler & Antonelli, 2014) theories from texts using external knowledge bases like HowNet and Wikipedia.

   3. Fuzzy logic-based approach: Sentence length, sentence similarity (Ropero et al., 2012), and other textual properties are inputs for the fuzzy logic technique that are later provided to the fuzzy system.

   4. Latent semantic analysis: The technique known as Latent Semantic Analysis (LSA) (Ozsoy, Alpaslan & Cicekli, 2011) allows text summarizing tasks to extract latent semantic constructions of sentences and phrases.

2. Supervised learning: At the sentence level, techniques linked to supervised extractive summarization are based on a classification strategy (Wikipedia, 2022). The model is taught by using examples to distinguish between non-summary and summary phrases.

   1. Machine learning depends on the Bayes rule: The machine learning method sees text summarization as a classification problem (Brownlee, 2019). The sentences are limited to non-summary or summary based on each attribute.

   2. Neural network based: It considers a RankNet-trained neural network with a two-layer and backpropagation approach (Kamper et al., 2015). To score the sentences in the document, the neural network system must first perform feature extraction on sentences in the test and training sets. This is done in the first phase, which uses a machine-learning approach for labelling the training data.

   3. Conditional random fields: A statistical modelling strategy called conditional random fields (Macherla, 2020) focuses on using machine learning to produce structured predictions.

1. Structure-based approach: It utilizes deep learning algorithms to choose the crucial passages from the original documents (Garg & Saini, 2019).

   1. Summarization based on tree method: (Kikuchi et al., 2014) uses a dependency tree to describe the text and information from the source text.

   2. Summarization based on template method: It is a method that gives the end user the freedom to design a template for the information that should be in summary (Oya et al., 2014). The template includes POS markers like adverbs, verbs, and nouns, among others, and the end user may define the method by which the sentences should appear in summary.

   3. Ontology-based method: The method for developing ontologies (Jishma Mohan et al., 2016) uses data preprocessing, semantic information extraction, and ontology development.

   4. Lead and body phrased method: It depends on the “insert and replace” process, which uses core sentences to replace the leading phrase and comparable syntactic head chunks at the beginning of each step (Sciforce, 2019).

   5. Rule-based method: Using this method (Vodolazova & Lloret, 2019), the textual materials are condensed by being shown as a collection of specifics.

2. Semantic-based approach: (Shahzad et al., 2022) In the semantic-based approach, ideas relevant to phenotypes are taken from the domain knowledge base’s class hierarchy and a semantic similarity metric determines their significance.

   1. Multimodal semantic model: In this method (Chen & Zhuge, 2018), the subject (images and manuscript data) of one or more documents is represented by a semantic unit that extracts the subject content and correlations among the topics.

   2. Information item based: Using the original text’s sentences as a starting point, Using this method, the original text’s abstract representation is used to construct the data for the summary.

   3. Semantic graph based: The Rich Semantic Graph (RSG) builds a semantic graph on the source content, condenses the semantic network, and then provides an exhaustive abstractive summary from the condensed semantic graph.

Problem statement and motivation

Considering advancements in software and hardware technologies and the expanded use of machine learning models, text summarization has changed over the past ten years as an NLP application (Yao et al., 2020). Extractive and abstractive summarization are the two basic technical subcategories of text summarization. Despite the vast quantity of information in Urdu web papers, there are many issues with text summaries in the Urdu language. To increase the readability of the Urdu language, it needs to generate a summary that retains the original text’s meaning. For Urdu languages, only extractive summarization is done by various algorithms and models, but not abstractive summarization.

Since abstractive summarizing of Urdu text is more difficult than extractive summarization, this paper’s primary contribution is the suggestion of an abstractive paradigm for summarizing Urdu texts. And improve the results for Persian language abstractive summarization. Due to the resulting summary’s abstractive character and Urdu’s intricate morphology, creating such a model is challenging. The suggested model was confronted with two primary obstacles: the first obstacle was the dispersed interpretation of the writing that considered the complexity of the Urdu language. The second problem was finding appropriate assessment metrics to judge the result of the generated summary.

Dataset

To conduct this research 1 Million news dataset is used, categorized into four types: Sports, Science & Technology, Business & Economics and Entertainment. It consists of long news text stories and their short summaries. The dataset incorporates news to date, URL, web source and the number of characters in news stories. We have used 70% of the data for training the model and the rest of the 30% for testing. It is the largest dataset in the Urdu language for performing natural language processing tasks.

Pre-processing

Typically, text preprocessing comes first in any NLP task. For the English language, a variety of open-source programs are available. It is still very difficult when it comes to Urdu or any other related languages, including Persian, Arabic, and Pashto. However several e-libraries are available for the Urdu language, but due to their poor accuracy, preprocessing still requires a lot of work. Normalizing textual texts is an important step in preprocessing. For instance, several nouns in the Urdu language contain digraphs, such as (( Alif (’ (’and Hamza (” ’ )) which, while being separate alphabets, have been employed as a separate notes. Separating these two letters is necessary for further processing. With relation to the syntactic organization, Urdu is a rich language. Several words may be written with or without a space. It is ensured that there are suitable gaps between words and pronunciations; The content normalization module will also remove diacritics and accents. It is converting a phrase into different forms, such as a list of tuples or a list of words, each of which has a form (word, tag). One of the crucial steps in-text pre-treating is tokenization. Tokenization can be considered as breaking down a text into specific or different terms, whether it sentences, phrases, paragraphs, or the complete document, as shown in Fig. 5. Tokenization assists in interpreting the text’s meaning by looking at the word order.

Tokenization involves breaking up long statements into sentences and words in two fundamental phases. By looking for word and sentence endings, tokenization of phrases is produced. These word counts are used as the start and stop positions for words. Exclamation points (!), question marks (?), and full stops (./-) are used as delimiters to separate paragraphs. Further processing, like lemmatization, is carried out based on these tokens. Lemmatization aims to break down words into their most fundamental components. As an illustration, the word ” ” ” is ” ” has the origin ”.” During the lemmatization procedure, the prefixes and suffixes are shortened, leaving only the word’s stem, As depicted in Table 1. Another crucial stage in text preparation is lemmatization. It is important to understand the context in which the word is used.

Stop words are defined as having significance in the context of semantic values. Removing these terms makes our text more focused on the key information by eliminating the low-level information. Only grammatically constrained orders to utilize these words. Stop words frequently appear in documents and their occurrence in phrases has little semantic significance. These words encompass a significant collection of archives without any meaningful value. The stop words, thus, for better language description, should be removed.

Urdu common stop words are . The content words (tokens) left after the stop-words are eliminated and then are available for processing as shown in Table 2.

Extractive summaries

After the pre-initialization operation is finished, the text’s characteristics are extracted. Extractive summarization is the process through which key phrases founded on a benchmark are chosen to give a comprehensive summary to communicate the original text’s key concept accurately.

Table 1:

Some extracted suffix.

DOI: 10.7717/peerjcs.1176/table-1

Table 2:

Sentences with and without stop words.

DOI: 10.7717/peerjcs.1176/table-2

Abstractive summary

For most NLP applications that use data sequences, such as machine translation and text summarization, the Seq2Seq recurrent neural network (RNN) architecture recently rose to the top. The encoder–decoder paradigm is employed when one sequence serves as the contribution and another as the output.

The vanishing gradient problem, which may be fixed by employing the LSTM, may affect the RNN. In this study, we’ll employ a model made up of an encoder–decoder LSTM. On the other hand, the proposed method utilised a multilayer encoder instead of a single-layer encoder. Three hidden state layers make up the multilayer encoder: the input text’s secreted conditions are on the top layer, the text’s secret states of its keywords are on the bottom layer, and the text’s hidden states of its name entities are on the top layer. The inputs for the three layers are word embedding of the text words, keywords, and name entities. For creating word vectors, 128 dimensions were considered as depicted in Fig. 6. Bi-directional LSTM units make up the three encoder layers’ hidden states. The input text order {tx = tx₁, tx₂, tx₃, …, tx_n} is generated by the first layer from right to left and is drawn to the hidden states {hs = hs₁, hs₂, …, hs_n,} correspondingly. The hidden states {h_k = hk1, hk2, hk3, …, hkq} are formed in the second layer and correspond to the depiction of the documents {k = k1, k2, k3, …, kq}. The keywords representation kv is created by concatenating the last forward keywords’ hidden state and the last backward keywords’ hidden state. The text name entities {nne = ne1, ne2, ne3, …, nee} are present in the text and provided as input to the hidden states hne = hne1, hne2, hne3, …, nee are represented in the final layer. To create the name entity representation nev, the most recent forward hidden state and the most recent backward hidden state are concatenated. In contrast, the decoder’s hidden states are made up of a single layer of a unidirectional LSTM. The decoder gets the word embedding produced by the decoder for the preceding word.

For the Persian language (Liaqat & Hamid, 0000), our critical commitment is that we extensively experiment with many things with various settings to combine BERT, GPT, and RoBERTa pre-prepared stations before launching our model, which is based on the transformer.

The models periodically report significant improvements on initial prototypes that use managed pre-planned models. More importantly, this simple process yields brand-new top-of-the-class outcomes in machine analysis, note synopses, phrase delivery, and sentence combining. The results of the suggested technique also demonstrate that a trained encoder is an essential component of arranging assignments. These tasks frequently profit from distributing the load across the encoder and the decoder. This study used more than 300 tests and several TPU v3 h in total, which increased the likelihood that these text-age-ready models’ language-displaying and cognition abilities would alter. This study ensures that NLP analysts and experts will gather useful information from the suggested outcomes as they take on the various seq2seq tasks. Using Word Piece, this research matched its content to the pre-prepared jargon of BERT, as seen in Fig. 7. The data collected from various resources is shown in Table 3 and the training time is illustrated in Table 4.