Automatic computer science domain multiple-choice questions generation based on informative sentences

Summarization

Text summarization methods belong to two types, abstractive and extractive. Abstractive summarization is closest to the way humans generate a summary. Abstractive summarization usually extracts the text’s key points and rephrases it, including the vocabulary beyond the specified text, and it is smaller in size (Genest & Lapalme, 2011). Undoubtedly, many researchers are working on abstractive summarization as it is closer to the human way of summarization, which is advantageous. But it requires a massive human summarization dataset for complex algorithms and deep learning, rules with restricted generalizability, and training of several GPUs over many days for automatic generation of summery. At the same time, extractive summarization creates a summary containing actual phrases and the same sentence structure from the source data. The proposed system only considers such text-based stems mined through extractive text summarization.

Quality phrases

A phrase contains more information than a word, so our system requires a phrase mining technique to extract quality phrases from the underlying domain. Similar works have been done previously using N-gram (Tahir et al., 2021) and topical phrase mining techniques. In detail, the N-gram technique facilitates identification and extracts frequent N-grams from the given text (Tahir et al., 2021). Similarly, topical phrase mining is a helpful technique for phrase mining, topic identification, social event discovery, etc. (Li et al., 2018). Another work proposes a knowledge discovery method for an information retrieval system to extract informative and most frequent phrases (Aljuaid et al., 2021). In Papasalouros, Kanaris & Kotis (2008), the researchers employ the online frequent sequence discovery method to extract frequent phrases. TF-IDF (Wu et al., 2008) and KEA (Zhu et al., 2013) are text analysis techniques for calculating raw frequency in a given text corpus. But these techniques calculate the raw frequency of phrases based on frequent pattern mining without considering their semantic meanings. For the extraction of quality phrases based stems from the extractive summary, a quality phrase mining technique (Liu et al., 2015) extracts semantically meaningful phrases instead of frequent patterns from raw text.

The quality phrase mining technique counts the word’s frequency, analyzes phrases semantically, and recognizes the quality phrases by considering some features. For example, quality phrase mining techniques make decisions based on features like concordance, completeness, informativeness, and popularity (Liu et al., 2015).

Introduction to WordNet

A WordNet is a machine-readable dictionary. It is a lexical database for the English language. In WordNet, nouns, verbs, adjectives, and adverbs are grouped into synonyms set. That set is known as synsets. Each word expresses a distinct concept. These synsets are interlined. There exist semantic relations and linguistic relations between the items of synsets. It works like a thesaurus, but WordNet has an advantage as it groups words with a particular sense. For example, wordNet lexicalized the main concept of “key” by making a synonym set of terms related to the idea. Figure 3 demonstrates an example of a concept hierarchy made by WordNet.

Wiktionary

Wiktionary is a web-based project it provides a free dictionary related to the content of terms. It contains data in a semi-structured form. In NLP tasks, Wiktionary offers the opportunity to convert lexicographic data into a machine-readable format.

Google search results

Google provides search results related to keywords given by the user in a query. The searched results are usually retrieved based on tokens in the query. The most relevant keyword appears at the top of the page. At the same time, they are arranged in descending order of relevancy. In creating the distractor list, the keywords of Google search results are also used.

Summarization

For sentence selection, some authors used the summarization technique like (Kurtasov, 2013) and extensive summarizer (Narendra, Agarwal & Shah, 2013), but the methods are outdated now; they did not implement deep learning in summarization. Until recently, recurrent neural networks (RNN) with long short-term memory (LSTM) have been used for many NLP tasks. But these methods do not perform well in the case of lengthy sequences and are prone to overfit even after a lot of tanning, massive computer resources, and many hours of training (Vaswani et al., 2017). Keeping this fact in mind, a better architecture known as Transformer is presented (Vaswani et al., 2017). The Transformer architecture is built using the attention mechanism and feed-forward neural networks (Vaswani et al., 2017). Though, Transformer overcomes many problems that occur while using RNN and LSTM. Later on, at the end of 2018, an unsupervised learning architecture BERT is presented at the top of Transformer architecture (Devlin et al., 2018). BERT is a trained architecture developed by researchers from Google. BERT surpasses several state-of-the-art methods in terms of performance for NLP tasks (Devlin et al., 2018). Since it is pre-trained architecture, it may be used for transfer learning to perform many NLP-related tasks (Barouni-Ebarhimi & Ghorbani, 2007). Prevalent methods lack to provide a dynamically sized summary. But using BERT, a dynamically sized summary may be generated since BERT generates sentence embeddings.

BERT for text embedding

As BERT gives outstanding performance than other NLP algorithms, BERT is selected for creating Text Embeddings. BERT is built on Transformer architecture, but its objectives are definite for pre-training. BERT masks out 10% to 15% of random words in the training data; masked words are attempted to be predicted. BERT also takes an input sentence and the candidate sentence to predict whether the candidate sentence follows the input sentence properly (Devlin et al., 2018). This training process is time-taking and requires a lot of computation power. It requires GPUs for training. Keeping this problem for public use, Google released two BERT models. One of these models comprises of 110 million parameters, while the second model includes 340 million parameters (Li et al., 2018; Aljuaid et al., 2021). As the large pre-trained BERT model provides outclass performance, so it is selected for summarization purposes. Using pre-trained BERT, Multiple layers may be chosen for creating embeddings. By using the [cls] layer, the NxM matrix is formed by BERT for clustering purposes. NxM refers to the number of sentences, and M represents embeddings dimensions. It is noticed that embedding representation produced by the [cls] layer is not nearly good. But due to BERT architecture, the tokenized words are equalized by creating NxExM embeddings for the output of other layers in the network, where M in NxExM balanced tokenized words. This issue can be overcome by averaging the embeddings to produce a matrix of order NxE.

Clustering embeddings

After completing the embedding from the N-2 layer, the matrix of order NxE is ready for clustering. Sciket learn library is used for the implementation of K-Means clustering. Sentences closest to the centroid are selected as candidate summary sentences.

Considering the background and related work, the gap in existing research and projects was automatic MCQ generation based on the informative sentence. An informative sentence is based on the topic’s core idea or meaningful sentences are extracted by the summarization technique by leveraging the most up-to-date BERT architecture.

Keyphrase extraction algorithm

Identifying noun phrases is an essential goal of the keyphrase extraction algorithm (KEA). In-text summarization, text categorization, and information retrieval (IR) systems, KEA plays an important role. KEA identifies phrases based on features like the position of the phrase in the document and the number of occurrences in the document. KEA also classifies the candidate phrase in the document. However, this algorithm sometimes provides incoherent phrases that don’t correspond to the text’s summary (Mishra & Singh, 2011). The KEA is used for both supervised and unsupervised ML tasks (Zhu et al., 2013). To mine high-quality N keyphrase candidates from the given text document, the KeyRank method is presented by Wang, Sheng & Hu (2017). KeyRank algorithm first identifies all phrases from the text. Subsequently, it ranks these candidate phrases, and finally, top-10 N key phrases are extracted from the document. Unlike KEA, KeyRank algorithm performs outclass and provides good results (Wang, Sheng & Hu, 2017). The key phrase extraction algorithm also effectively identifies the number of clusters in a massive number of documents (Han, Kim & Choi, 2007). To mine top-quality keyphrases from the text, Wang, Mu & Fang (2008) presented a technique. The technique was based on the semantic meaning of phrases. The method first extracts the phrases from text and then the semantics of these phrases are checked using the word sense disambiguation technique.

TF-IDF

TF-IDF refers to term frequency, inverse document frequency. As the name depicts, this method calculates the relevance of terms in specific documents (Qaiser & Ali, 2018). TF calculates the raw frequency of terms by considering the number of occurrences of that word in the document. In comparison, IDF assigns the weights to the words. To remove stop words, lower weights are assigned to highly frequent terms, and higher weight is given to low frequent words (Sohaib & Olszak, 2021). The attention-based refined TF-IDF method was proposed by Zhu et al. (2019), which identifies hot terms in the document based on time distribution information. TF-IDF is considered as an essential practice for discovering hot terms within the document (Zhu et al., 2019).

Evaluation of stem and key

It is observed that the majority of MCQ systems are evaluated by human evaluators as there is no standard dataset publicly available for evaluating automated generated MCQs. For evaluations, evaluators created private data for testing system quality with human evaluators’ help. For stem and key evaluation, sentence length, simplicity of sentence, the difficulty of sentence and key, informativeness of sentence, the sufficiency of context, the difficulty of the key, domain relevancy, grammatical correctness, and correctness of sentence have been used in evaluation metrics. In Table 1, an overview of different systems is presented. The accuracy of the systems is not compared due to different approaches and the unavailability of a benchmark dataset (Shah, Shah & Kurup, 2017; Susanti et al., 2017).

Evaluation of distractors

The metrics used by various researchers to evaluate distractors are difficulty, readability, closeness to key, and usefulness. Pino, Heilman & Eskenazi (2008) distractors being assessed based on semantic and syntactic points of view. Teo (2020) tested readability and semantic meaning of distractors by substituting the distractors in the gap. Bhatia, Kirti & Saha (2013) defined the scale “good” if at least one of the distractors is close to the key. Table 1 shows the type of evaluation, evaluation metrics adopted by various researchers, and their systems’ accuracy.

Summarization module

The summarization module is responsible for creating an extractive summary of input data. In this way, the original text may be reduced up to 50%. In addition, the summary module discards insignificant lines of the input text. We employed a deep learning-based approach, BERT, which creates sentence embeddings for subsequent clustering by employing the K-Means. More precisely, raw text is passed to the BERT model for creating sentence embeddings. The embeddings are then clustered using K-Means clustering, then sentences closest to the centroid are selected as candidate summary sentences. The candidate summary sentences are then scored based on features; the sentences with a high score are chosen for the MCQ generation process.

Scoring module

The summarization module creates an extractive summary of input data, thus reducing around 50% of the input text. The output is then given to the scoring module. The scoring module scores all the candidate sentences of summary. The scoring is done based on the features listed in Table 2.

Quality phrases

The quality phrase mining algorithm is partially automated. It needs to be trained first to generate quality phrases automatically. The computer science domain’s quality phrases are collected from a dataset and verified by domain experts. Figure 8 shows the parameters of quality phrases.

A phrase is considered as a quality phrase if it possesses the properties of informativeness, completeness, concordance, and popularity (Liu et al., 2015). In addition, a quality phrase must be frequent in a given corpus; it cannot be a quality phrase if it is not frequent. A phrase’s quality can be defined as the likelihood of a multi-word series comprising logical and consistent semantic meanings. Suppose if v is a phrase, then Eq. (1) shows the formulation to calculate the quality of phrases.

(1) $Q(v)=p(\lceil v\rfloor |v)\epsilon [0,1],$where $\lceil v\rfloor$ signifies the occurrence of a word in v making up a phrase. If a word is distinct, its quality would be Q(w) = 1. The values between 0 and 1 estimate the phrase or phrase quality (Liu et al., 2015). Figure 9 shows a detailed procedure of quality phrase extraction from the input text.

The details of the features of quality phrases are briefed as follows:

Popularity

A quality phrase should frequently appear in the whole given text. A phrase cannot be a quality phrase if it is not occurring with sufficient frequency.

Concordance

Identify the sentences with similar meanings. Identification of synonyms is also essential. Equation (2) is used to calculate the concordance feature.

(2) $$p\left(u\right)={\displaystyle \frac{f\left[u\right]}{\sum _{u^{\mathrm{\prime }}\epsilon u}f\left[u^{\mathrm{\prime }}\right]},}$$where u is a word or phrase and u ε U, f[u] is raw frequency, then p(u) shows its probability (Liu et al., 2015).

Informativeness

A phrase is informative if it gives information about a specific topic. Equation (3) presents the formula for the calculation of informativeness.

(3) $$IDF\left(w\right)=\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\left|C\right|}{\left|\left\{d\epsilon \left[D\right]:w\epsilon C_d\right\}\right|}}$$

Equation (3) calculates the Average inverse frequency of the document. This equation |C| represents corpus, d represents a document, and w represents words in a given document (Liu et al., 2015).

Completeness

A phrase is said to be complete if it gives full semantic meanings regarding a specific context. Therefore, a phrase should possess completeness to be a quality phrase.

TF-IDF

This is another scoring feature used for the proposed system. TF-IDF provides the weight of every word in the given text. This method is numerical statistics (Qaiser & Ali, 2018). TF is the term frequency, which shows how many times a term occurs in a document. IDF gives lesser weight to repeated words and high weight to rare words to remove the stop words from the specified documents (Sohaib & Olszak, 2021). The mathematical Eqs. (4)–(6) are used to compute TF-IDF.

(4) $$t{f}_{i,j}={\displaystyle \frac{n_{i,j}}{\sum _k{n}_{i,j}}}$$

Equation (4) is used for the calculation of the term frequency of words. In this equation $t{f}_{i,j}$ corresponds to term frequency, of i in j (http://www.tfidf.com/); where i is the word whose frequency is to be computed and j is the number of documents having i, $n_{i,j}$ is the number of instances word i emerges in the document and $\sum _k{n}_{i,j}$ represents the total number of words in the text.

(5) $$Idf(w)=log({\displaystyle \frac{N}{d{f}_t})}$$

Equation (5) is used to analyze the inverse document frequency of the terms. N is the overall number of documents, $d{f}_t$ is the amount of documents containing the term t.

(6) $$Wi,j=t{f}_{i,j}\ast \mathrm{l}\mathrm{o}\mathrm{g}({\displaystyle \frac{N}{d{f}_t})}$$

Equation (6) is used to compute the weight of each word in the document. To calculate the weight of words within the text, term frequency multiplies with the inverse document frequency.

No. of nouns and verbs

The number of nouns shows the # of nouns and verbs in the sentence.

No. of stop words

It shows the number of stop words in the sentence.

Jaccard similarity of the title with sentences

The Jaccard similarity of the title of the chapter is compared with each sentence. A sentence having tokens similar to the title gets more scores. Equation (7) depicts the formulation of the Jaccard similarity index.

(7) $$J\left(A,B\right)={\displaystyle \frac{\left|A\cap B\right|}{\left|A\upsilon B\right|}={\displaystyle \frac{\left|A\cap B\right|}{\left|A\right|+\left|B\right|-\left|A\cap B\right|},}}$$where A is the set of tokens in the title, while B contains tokens in the sentence.

Each candidate sentence is scored based on scoring features, and a total score (sum of all the scores) is assigned to each sentence as presented in Eq. (8). The output is then sent to the stem selection module for further processing.

(8) $$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\mathrm{Q}\mathrm{P}\mathrm{S}+\mathrm{T}\mathrm{F}-\mathrm{I}\mathrm{D}\mathrm{F}+\mathrm{C}\mathrm{N}\mathrm{V}+\mathrm{C}\mathrm{S}\mathrm{W}+\mathrm{J}\mathrm{S},$$where, QPS referes to quality phrase score, TF-IDF depicts term frequency-inverse document frequency score, CNV denotes.

Sentence selection module

The scoring module assigns a score to each candidate sentence, and the output proceeds to the sentence selection module. Finally, the stem selection module is responsible for selecting top-ranked sentences. This module chooses 20% of top-ranked sentences as candidate sentences for MCQ generation. The selection is made on the aggregated/total score of the sentence. Hence, informative sentences are extracted.

Key selection

It is noticed that the selection of keys relevant to the educational context is dependent on human judgment. A dictionary contacting pertinent keys to the computer science domain is built. The key dataset is used for picking the important keyword as a key from a sentence. A dataset of 1,327 keys is made for the key selection module using 9th and 11th-grade books and technology websites (Baig, 2018; Chattha, 2019; https://www.computerhope.com/jargon.htm; https://www.techopedia.com/dictionary). At this step, the system selects a key from the candidate sentence by

• Skimming the sentence

• Finding domain-relevant keys in the sentence with the help of the dataset.

Stem formation

Once the key is selected, the stem can be formed by replacing the key with a blank. The underlying module works on the following steps:

1. Scan sentence

2. Select key

3. Replace the key with fill in the blank

Distractor generation

The key is given as a sample of input text to the distractor generator for distractor generation. Next, a list of distractors is created using WordNet, Wiktionary, and Google search results. This process includes the following steps for distractor generation.

Creating a list of distractors

• By using WordNet, finding synonyms of key

• Finding synonyms on Wiktionary one by one

• Providing derived words of key

• Repeating the process for all synonyms

• Adding all results in a list of dictionary

• Finding list items on Google search

• Picking one item from the list

• Including the “AND” operator as a search query, i.e., “Keyboard And.”

• Searching given suggested Google equery one by one

• Scanning results of the searched query for relevant keywords

• Adding discrete effects in a list of dictionary

The system uses online knowledge databases and resources available on the internet for candidate distractor generation. The first step for distractor generation is finding the key synonyms and finding similar or relevant keywords. Three different candidate distractors include synonyms containing a similar or related concept pertinent to the correct answer. Next, network-accessible encyclopedias like Wiktionary and Google search results are used for selecting distractors. After generating the list of distractors, the next step is the selection of distractors. Then, randomly three distractors are chosen from the list. The distractor dictionary diagram is shown in Fig. 10.

Front end

A desktop application is designed that provides an integrated development environment to the user. The user inputs a raw text for the interface, and as a result, the output is shown on the desktop app, and a “.txt” file is also created containing all the output of the given input, i.e., MCQs.

Back end

All the backend files of the system reside on cloud service. The desktop application retrieves the functionality of the backend system and provides the results to the user. System architecture components are presented in Fig. 11.

System requirements

Automatic MCQ generation system based on two components front end and back end. The system’s back end resides on the AWS cloud, while the front end comprises of a desktop application that runs on the user terminal. The specifications of the server and terminal are given in Table 3.

Advantages

Machine deep learning techniques used in the system help to achieve the followings:

• Fast and efficient results

• Free of bulky computation devices

• Bettering learning process

• Easily accessible

Achieving research objectives

• We can reduce the research gap

• MCQs are based on informative sentences

• These reduce the cost and time of finding informative sentences, keys, and appropriate distractors.

Assumption

An assumption must be considered along with the improvement of the proposed system. First, a high-speed internet connection is required for using the system smoothly without any disturbance.

Future work

This system may further be improved by introducing abstractive summarization techniques. This system can also make MCQs of other domains by enhancing the dataset of keys and quality phrases. The method for distractor selection may be improved to make more confusing or difficult distractors. Further work can be done on the front end of this system by providing options for the number of required MCQs.