A literature review of research on question generation in education

Rational

Regarding QG in education (QGEd), there are also some relevant reviews. We found that some reviews did not explicitly propose the characteristics and objectives of QGEd, or they were not sufficiently focused on the educational field. For instance, Le, Kojiri & Pinkwart (2014) reviewed the research progress of QGEd, categorizing it into three types from the perspective of application objectives: knowledge/skill acquisition, knowledge evaluation, and tutoring dialogues. Kurdi et al. (2020) also provided a review of this field, but there was insufficient discussion on the guidance and integration of educational theories in generating questions. In 2024, Al Faraby, Adiwijaya & Romadhony (2024) conducted a review of QGEd from 2016 to the beginning of 2022, focusing on analyzing the technical components of QG, such as context, answers, questions, and datasets. Although this study concentrated on reviewing methods based on neural network models and involved a relatively small number of literatures, it clearly proposed that naturalness, diversity, controllability, practicality (educational value), and personalization are the core objectives of research in this field, providing a clear direction for QGEd research:

• (1)

  Naturalness means that the generated questions should not only conform to the expression habits and grammatical rules of the target language but also ensure that the questions can be answered from the context or at least can be resolved using background knowledge related to the context. This goal is a universal requirement for all QG tasks, not limited to QGEd.

• (2)

  Diversity refers to generating a variety of different types of questions for the same context to avoid producing overly general and similar questions, in order to meet the needs of different teaching scenarios.

• (3)

  Controllability mainly points to the cognitive level and difficulty level of the generated questions, that is, guiding the model to generate questions of different cognitive levels or difficulty levels through corresponding mechanisms.

• (4)

  Educational value emphasizes that the generated questions should focus on teaching objectives and content, avoiding the generation of some popularized and trivial questions.

• (5)

  Personalization aims to meet the individual differences of learners and specific educational needs.

Among these objectives, naturalness has been well addressed in previous research. For instance, in studies related to heuristic rule-based methods, language features are often utilized, such as named entity recognition (Sang & Meulder, 2003), positional tagging (Lindberg et al., 2013), or by using placeholders (Jia et al., 2021; Kumar, Banchs & D’Haro, 2015; Olney, Pavlik & Maass, 2017) to replace keywords or sentences in the context; in neural question generation (NQG) methods, many works have improved the quality of questions by refining the encoding and decoding processes (Wang et al., 2019; Song et al., 2018; Kim et al., 2019). Furthermore, some studies have adopted strategies such as question ranking (Zheng et al., 2018), reinforcement learning (Fan et al., 2018), and multi-task learning (Wang, Yuan & Trischler, 2017) to further enhance the naturalness of the generated questions. And personalization is a comprehensive embodiment of other objectives. In summary, this study believes that diversification, controllability, and focus on teaching content should be the core objectives of QGEd.

In terms of literature breadth, this review covers more QGEd research articles than previous reviews, totaling 48. Regarding the review’s scope, it is more comprehensive and systematic than previous work, covering the development of QG core technologies, the current state of research on QGEd’s three core objectives, datasets, question evaluation schemes, and future research directions. In terms of perspective, this review starts from educational theories and practical needs, deeply discussing the achievements and shortcomings of current technical research, thereby providing better guidance for its future development. In summary, this review introduces existing QGEd technologies developed by other researchers, defines their core algorithms and distinctive features, and helps researchers gain a thorough understanding of the current state of QGEd development, thereby inspiring them to take the next steps in their work.

Intended audiences

This review is intended for two main groups of audiences with varying levels of expertise in computer. The first group consists of non-technical individuals, such as teachers and educational researchers, who may not have extensive experience with computer techniques. The second group includes technical audiences, such as education technology researchers and developers in the artificial intelligence industry, who may have more familiarity with these techniques. The first objective is to motivate teachers and educational researchers who have relied heavily on conventional question design and lack an understanding of computer vision algorithms. We aim to encourage them to engage with question design used in practical teaching and teaching research. The second objective is to assist and inform education technology researchers and developers about the current state of QGEd. This allows them to take inspiration from existing work and make modifications, thereby preventing them from having to reinvent the wheel.

The rest of this article is organized as follows. The methodology used to conduct this survey is discussed in ‘Survey Methodology’. The progress of QG from the perspective of technological evolution and the evolution trends of QGEd are introduced in ‘Development of Technology’. The research progress of QGEd in achieving the three core objectives is discussed in ‘Three Core Objectives’. An overview of typical datasets commonly used for QG is presented from the perspective of QGEd in ‘Datasets’, followed by a review of commonly used evaluation schemes for QGEd in ‘Evaluation Schemes’. Finally, discussions and future work directions are presented in ‘Discussions’, and the overall conclusions are summarized in ‘Conclusion’.

Information source and search process

The five scientific databases were considered during the initial phase (see Fig. 1): Web of Science, IEEE Explore Digital Library, Springer Link, DBLP Computer Science Bibliography and ACM Digital Library. Each database includes relevant studies about QGEd. The search strategy included limiting the search results to studies that were published between 2015 and 2025 and that were written in English. The keywords chosen to identify the relevant records were “question generation,” “question generator,” “question generation for education,” “difficulty-controllable generation,” “question-type driven question generation,” “diverse question generation,” “Automatic education QG.” It should be noted that this review primarily focuses on studies related to English QG, as the majority of the literature retrieved from citation databases centers on English. However, this does not imply that research on QG in other languages is excluded, which may also provide valuable insights and directions for future exploration.

Articles selection

The initial search on those five databases returned 590 articles. The titles and summaries of these articles were screened to exclude irrelevant works. As a result of this first screening, 208 articles were selected and considered for the next screening phase, which included a full review of the articles to check whether the eligibility criteria were met. Finally, 48 unique and fully accessible studies were identified as primary sources for further analysis in this review. Details about the number of records retrieved by each scientific database and the process of the data extraction and monitoring are shown in Fig. 1.

Data items

Each selected articles were indexed in a local database and, for each study, the following characteristics were included: title, year, core technology (heuristic rule, neural network or the combination of these two), core objective, database, evaluation metrics and evaluation/assessment strategies. Such characteristics were deemed relevant to reach the aim of the present work.

Synthesis of results

The authors of this article, based on the selected articles, provides a detailed and comprehensive review of the technological development trajectory, the current status of research on the three core objectives, commonly used datasets in research, and evaluation schemes of QGEd in the rest of sections. The basic information of the selected articles is also organized in the Table 3 for readers to quickly and conveniently locate the reference literature that meets their needs.

Heuristic rule-based methods

Heuristic rule-based methods were the mainstream approach in early question generation. These methods typically rely on artificially designed transformation rules to convert given contexts into corresponding questions. They can generally be divided into three categories: template-based methods (Mostow & Chen, 2009; Zheng et al., 2018; Fan et al., 2018; Liu et al., 2019; Fabbri et al., 2020), grammar-based methods (Wolfe, 1976; Kunichika et al., 2004; Mitkov & Ha, 2003; Heilman & Smith, 2010; Ali, Chali & Hasan, 2010), and semantics-based methods (Huang & He, 2016; Yao & Zhang, 2010; Flor & Riordan, 2018; Dhole & Manning, 2020). Generally, these methods include two steps: content selection and question construction. First, the topic to be inquired about is selected through semantic or syntactic parsing methods, and then templates or transformation rules are used to convert the context of the selected topic into a natural language question.

Neural question generation

In 2017, Du & Cardie (2017) first applied the sequence-to-sequence (Seq2Seq) encoder-decoder model to question generation. Since then, neural question generation (NQG) has become the mainstream approach in QG research, leading to a surge of high-quality studies and achieving more desirable results. Most of the work in NQG utilizes recurrent neural networks (RNN) or transformer networks with attention mechanisms to automatically learn patterns and relationships in the input text, capturing complex dependencies and generating questions that may not be explicitly defined in rules, adapting to different domains when there is sufficient training data available.

The core technology evolution trends in QGEd

To obtain high-quality data for the research, this study utilizes widely recognized scientific literature databases, including Web of Science, ACM, dblp, IEEE, and Springer. The search period is limited to January 2015 to February 2025, and a total of 48 QGEd research articles were collected. Based on the publication years of the literature, QGEd experienced a research peak between 2017 and 2018, as shown in Fig. 2. During this period, the number of studies using deep neural network models peaked at 7; in contrast, the number of studies based on heuristic rules rapidly decreased. After 2020, heuristic rules were primarily used to address the limitations of neural network models in controlling generation factors. Moreover, in recent years, prompt engineering has emerged as a novel and highly efficient method for guiding LLMs to automatically generate questions.

Diversification

In the field of education, the diversification of question design is a widely recognized need. Diversified questions not only promote learners’ thinking at different cognitive levels and deepen their understanding of the learning content, but also stimulate learning interest and creativity, and cultivate a multi-perspective approach to problem-solving. In the field of QGEd, question diversification is mainly reflected in two aspects: expression diversification and question type diversification (Zhang & Zhu, 2021). Expression diversification refers to conveying the same semantics through different expressions, and these questions, although stated in various ways, all point to the same answer. This kind of diversification helps to cultivate and exercise students’ transferability and semantic discernment abilities. Question type diversification, on the other hand, involves the use of different types of questions, such as fill-in-the-blank, multiple-choice, short answer, and true/false questions, to meet different teaching needs and learners’ learning styles.

Controllability

Controllable QGEd primarily refers to the controllability of question difficulty and cognitive level. Difficulty and cognitive level are two interrelated but independent concepts. In the teaching process, it is necessary to design questions of different cognitive levels for the learning content based on teaching objectives, and different cognitive levels reflect different question difficulties. On the other hand, due to the varying actual ability levels of learners, the perceived difficulty of the same question may differ. Therefore, when generating questions, it is important to consider not only the cognitive level involved in the question but also the difficulty of the question.

Focus on teaching content

A question may belong to a higher cognitive level, but this does not automatically endow it with educational value. Questions with educational value should enable learners to gain knowledge, develop skills, and achieve teaching objectives or fulfill the purposes of teaching evaluation in the process of answering the questions. To generate questions with educational value, and to make the generated questions more educationally valuable, some studies focus on generating questions targeted at content with educational value within the context, and use richer information related to the answers as input to improve the quality of question generation, such as the external knowledge background of the answers, semantic features, and positional features.

Regarding the introduction of external knowledge background information, Liu & Zhang (2022) utilized the Chinese Wikipedia knowledge graph (CN-DBpedia2) to extract knowledge triples related to the answers and used the pre-trained language model BERT to encode the target answers and the graph separately, enriching the answer feature information and enhancing the educational value of the generated questions. In terms of utilizing the semantic features of answers, Zhou et al. (2017) incorporated linguistic features such as part-of-speech, named entities, and word forms into the QG system’s input; Du & Cardie (2018) improved the accuracy of questions by annotating each pronoun’s antecedent and its coreference location features in the input text. Regarding the processing of answer location features, Sun et al. (2018) designed a position-aware model that enhances the model’s perception of the context word positions by encoding the relative distance between context words and answers as position embeddings; Cheng et al. (2021) utilized open information technology to construct the original text into a directed graph composed of entity nodes and relationship edges, and then encoded it with graph neural networks, thereby leveraging the positional information of the answer nodes within the graph.

However, the above technologies, which focus on teaching content, all adopt answer-aware QG strategies, requiring the target answers to be manually annotated in advance. This undoubtedly increases the workload of human teachers in practical applications, reflecting the issue of insufficient technological intelligence level. Therefore, technologies that can generate answers and corresponding questions based on a given paragraph have been valued by researchers. This technology is known as joint question-answer generation (Cui et al., 2021). In joint question-answer generation, automatically selecting content with educational value from a given context is a key step. For example, Chen, Yang & Gasevic (2019) proposed nine strategies for selecting important sentences, including choosing sentences at the beginning of a paragraph, sentences containing new words, and sentences with the lowest similarity to other sentences. Steuer et al. (2021) used keyword filtering and classifiers to filter out sentences containing key concepts, and then used dependency parsing and semantic graph matching techniques to identify and extract content with complete semantic information and educational value from textbooks as target answers for generating questions. Yao et al. (2022) used heuristic rules to select key words that help understand the story, such as characters, emotions, and causal relationships. Ch & Saha (2023) identified key content in textbooks through word frequency statistics. In addition to rule-based and statistical methods, there are also studies that train models to automatically select sentences. For example, Du & Cardie (2017) used a hierarchical neural network framework to encode the input context at both the sentence level and paragraph level, treating sentences containing the target answer as the target sentences, and the given context as input, directly training a neural network model capable of automatically extracting sentences with question-asking value.

Recently, Maity, Deroy & Sarkar (2025) directly utilized education-related content as the data source, integrating retrieval-augmented generation (RAG) based on BART with prompt engineering for large language models. By combining the strengths of retrieval and few-shot learning, it generated higher-quality questions that better aligned with educational objectives.

Non-education related datasets

The content of non-education-related datasets is primarily sourced from Wikipedia, news articles, speech transcripts, or paragraph texts from the internet, rather than educational textbooks. Even so, some datasets that encompass non-factual questions and difficulty classification features also offer valuable resources for exploring how to generate questions that are of higher cognitive levels or differentiated in difficulty. For instance, HotpotQA (Yang et al., 2018) includes two modes: simple and difficult, with solving difficult mode questions requiring cross-document reasoning. NQ (Kwiatkowski et al., 2019) is divided into “simple, medium, difficult” difficulty levels based on the naturalness of the questions and the findability of the answers. ODSQA (Lee et al., 2018) may require reasoning and understanding based on colloquial questions. The questions in CoQA (Reddy, Chen & Manning, 2019) are generated based on a segment of situational dialogue stories, forming a logically coherent sequence of questions. This type of question is particularly beneficial for cultivating learners’ language logical expression abilities. The questions in Cosmos QA (Huang et al., 2019) require common sense reading comprehension, mainly including questions about the possible causes or consequences of events.

Education related datasets

Education related datasets not only focus on content that can be used for education as their data source but also more universally pay attention to the collection and processing of questions with different difficulties and more cognitive levels. Among them, ARC (Clark et al., 2018), although it includes a variety of question types, only simply divides the questions into an Easy set and a Challenge set. RACE (Lai et al., 2017), as described in section “THREE CORE OBJECTIVES,” has achieved considerable results in various QGEd studies and is divided into two difficulty levels: the middle school RACE-M and the high school RACE-H. CLOTH (Xie et al., 2018) categorizes questions into four types based on the cognitive effort required to answer them: Grammar, Short-term reasoning, Matching/Paraphrasing, and Long-term reasoning, with the largest proportion of Long-term reasoning questions. FairytaleQA divides questions into explicit and implicit categories based on whether the answers can be directly found in the text. In fact, the former belongs to factual questions, whereas implicit questions are non-factual questions that require summarization and judgment based on implicit information in the text.

Due to the large scale and high-quality real exam questions of RACE, it subsequently became one of the most commonly used datasets in QGEd, and some follow-up work has further supplemented RACE. In 2019, Liang, Li & Yin (2019) released the RACE-C, in which they collected real exam questions from Chinese university English reading comprehension, further enriching the difficulty range of RACE’s questions. In 2021, based on RACE, Jia et al. (2021) released EQG-RACE specifically for educational QG tasks. EQG-RACE removed general questions not suitable for QG tasks and automatically annotated the key sentences of the questions by calculating the ROUGE-L between the answers and the article sentences. In 2023, Zyrianova, Kalpakchi & Boye (2023) conducted a detailed evaluation of RACE by scoring some data samples and manually annotated the key sentences of some questions, releasing EMBRACE.

Although these datasets have potential for QGEd research, the division of cognitive levels and difficulty in most datasets is empirical, lacking unified, clear, and quantifiable standards, which is not conducive to the development of controllable QGEd with cognitive levels and difficulty. To achieve more fine-grained controllable QGEd, a more specific and scientific classification and annotation of the cognitive levels and difficulty of questions are still needed. For example, LearningQ (Chen et al., 2018) uses the Bloom’s Taxonomy introduced in section “THREE CORE OBJECTIVES” to annotate and categorize questions, but only 200 random questions in the dataset have this classification label, which is far from meeting the training needs of NQG.

In fact, according to section “THREE CORE OBJECTIVES,” recent outstanding research in the field of QGEd has conducted extensive work on self-built datasets. These efforts essentially involve either automatic or manual annotation of training data with difficulty levels or cognitive hierarchies. This highlights the significant role of question difficulty and cognitive hierarchy, as two educational attributes, in the construction of datasets specifically dedicated to QGEd research.

Automatic schemes

Currently, most QGEd studies employ automatic evaluation metrics inherited from traditional QG research, with BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee & Lavie, 2005), chrF (Popović, 2015) and BERTScore (Zhang et al., 2020) being the primary ones.

• 1.

  BLEU

BLEU is an n-gram precision-based evaluation metric that evaluates quality of generated text by comparing the n-gram overlap between generated text and a set of reference texts, with the specific calculation method as shown:

(2) $$\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}=\mathrm{B}\mathrm{P}\cdot \exp (\sum _{n=1}^n{w}_n\log p_n).$$

Here, p_n represents the precision of the nth gram, w_n is the corresponding weight, which is typically a uniform weight in BLEU, meaning that the weight of each n-gram is the same. BP stands for the brevity penalty factor, which comes into effect when the length of the generated text is less than or equal to the length of the reference text.

• 2.

  ROUGE

ROUGE evaluates the quality of generated text by comparing the overlapping units (such as n-grams, word sequences, and word pairs) between machine-generated text and ideal reference texts. It includes a total of four metrics: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S. The value of ROUGE-N is obtained by dividing the number of shared n-grams by the total number of n-grams in the reference summary. This ratio reflects the similarity in content between the generated text and the reference text. The specific calculation method is shown as:

(3) $$\mathrm{R}\mathrm{O}\mathrm{U}\mathrm{G}\mathrm{E}-\mathrm{N}=\frac{\sum _{S\in \{RefrenceSummaries\}}\sum _{gra{m}_n}Coun{t}_{match}(gra{m}_n)}{\sum _{S\in \{RefrenceSummaries\}}\sum _{{gram}_n}Count(gra{m}_n)}.$$

This formula calculates the n-gram recall rate, which is the ratio of the number of common n-grams between the machine-generated text and the reference text to the total number of n-grams in the reference text.

ROUGE-L is an evaluation method based on the longest common subsequence (LCS). It seeks the longest sequence of identical words between two sequences (generated text and reference text). The longer the LCS, the more similar the two texts are. ROUGE-L uses the F-measure (a metric that combines precision and recall) to evaluate similarity. ROUGE-W is an evaluation method for weighted longest common subsequence (weighted LCS). It is similar to ROUGE-L but adds weighting for consecutive matching words, meaning that if two words are consecutively matched, they will receive a higher score. ROUGE-S is based on the statistics of Skip-Bigrams. Skip-Bigrams are pairs of words that appear in a sentence but can have any number of other words in between. In summary, ROUGE focuses more on the recall of correct information in the generated text compared to BLEU when evaluating the similarity between generated text and reference text, that is, how much information from the reference text is included in the generated text.

• 3.

  METEOR

METEOR is an evaluation metric that takes into account synonyms, morphological variations, and sentence structure. Its calculation is more complex, involving the alignment of words in the generated text and the reference text, and calculating a score based on matches. The calculation formula is shown as:

(4) $$\begin{array}{c}Fmean=\frac{10PR}{R+9P}\hfill \\ Penalty=0.5\ast {\left({\displaystyle \frac{\mathrm{\#}chunks}{\mathrm{\#}unigram{s}_{matched}}}\right)}^3\hfill \\ \mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=Fmean\ast (1-Penalty).\end{array}$$

METEOR can calculate the similarity between generated questions based on word-level precision and recall, as well as a penalty for word order. The Fmean is the harmonic mean of unigram precision P and unigram recall R, with a higher weight given to recall as 9R. The Penalty calculates the degree of fragmentation of matches due to word order disruption; if words in the generated text match with words in the reference text but are out of order, the Penalty increases.

• 4.

  ChrF

ChrF (Character n-gram F-score) is an evaluation metric based on character-level assessments, primarily used to evaluate the quality of text generation tasks. It assesses the similarity between reference and generated texts by calculating the matching counts of character-level n-grams. The core advantages of chrF include: (1) It can better capture syntactic and semantic information in morphologically rich languages; (2) It is language-independent, as it does not rely on language-specific tokenization processes and is applicable to multiple languages; (3) It is both simple and efficient, requiring no additional tools or knowledge sources. The general formula for the chrF score is:

(5) $$\mathrm{c}\mathrm{h}\mathrm{r}\mathrm{F}\beta =\left(1+{\beta }^2\right){\displaystyle \frac{\mathrm{c}\mathrm{h}\mathrm{r}\mathrm{P}\cdot \mathrm{c}\mathrm{h}\mathrm{r}\mathrm{R}}{{\beta }^2\cdot \mathrm{c}\mathrm{h}\mathrm{r}\mathrm{P}+\mathrm{c}\mathrm{h}\mathrm{r}\mathrm{R}}.}$$

Here, chrP and chrR represent the precision and recall of character n-grams, calculated as the average across all character n-grams. The parameter β is used to adjust the relative importance of recall compared to precision, specifically giving β times more weight to recall. This metric is language-independent and token-agnostic, making it highly versatile and widely applicable across various translation tasks.

• 5.

  BERTScore

BERTScore is a text similarity evaluation metric based on the BERT pre-trained model, designed to measure the semantic similarity between generated text and reference text. It leverages contextual embeddings, utilizing the context-aware word vectors generated by the BERT model to capture semantic information. This approach enables BERTScore to demonstrate strong semantic matching capabilities, outperforming traditional evaluation methods based on n-gram overlap (such as BLEU) in capturing semantic similarity. Additionally, BERTScore supports evaluation across multiple languages, further enhancing its applicability in diverse linguistic contexts.

The calculation process of BERTScore is as follows: (1) Encode the candidate and reference texts using the BERT model to obtain contextual embeddings for each word; (2) Calculate the cosine similarity between each word in the candidate text and each word in the reference text; (3) Use a greedy matching algorithm to find the most similar word in the reference text for each word in the candidate text; (4) calculate precision, recall, and F1-score based on the matching results. Precision quantifies the proportion of words in the generated text that match the reference text, while recall measures the proportion of words in the reference text that match the generated text. The F1-score serves as a balanced metric, taking into account both precision and recall.

• 6.

  Other metrics

The aforementioned several metrics are commonly used in traditional QG research to evaluate the natural fluency of generated questions, focusing on the relevance and answerability of the generated questions, but they cannot directly evaluate the cognitive levels, difficulty, and diversification of the questions, which are the quality dimensions of more concern in the field of education. Researchers, considering this, have begun to explore other metrics for automatically evaluating the quality of generated questions. In terms of evaluating the cognitive level of questions, Stasaski et al. (2021) designed the Cause/Effect Presence metric to specifically evaluate the quality of causal questions. When the expected answer to a generated question is a “cause,” the Cause Presence metric checks whether the “effect” extracted from the text is mentioned in the question. For questions where the “cause” is the expected answer, the size of the intersection between the words of the extracted “effect” appearing in the question and the set of “cause” words is calculated, then divided by the size of the “cause” word set to obtain the cause recall, and the same goes for Effect Presence. Steuer, Filighera & Rensing (2020) used the frequency of Wh-words to make a rough evaluation of the cognitive level of questions.

For automatic difficulty evaluation, please refer to section “Difficulty-controllable QGEd,” which can be divided into absolute difficulty evaluation (Gao et al., 2019; Srivastava & Goodman, 2021) that only considers the complexity of the question itself, and relative difficulty evaluation (Bi et al., 2024; Cheng et al., 2021) that is calculated in combination with the test-taker’s ability.

It is mainly evaluated by inversely evaluating similarity to evaluate the diversification of the questions, that is, the degree of difference in expression. For example, if a generated question is not similar to any question in the reference set, it can be considered novel. Srivastava & Goodman (2021) used novelty to evaluate question quality, which essentially uses text similarity metrics (such as cosine similarity, Jaccard similarity, etc.) to evaluate the difference between each generated question and the questions in the reference set. Wang et al. (2020a) and Gou et al. (2023) both used the pairwise metric to calculate the average BLEU-4 score between any two questions in the generated question set. A lower pairwise indicates a higher degree of question diversity.

In summary, automatic schemes focus on evaluating the similarity between the generated questions and the target questions, while relying on manual evaluation for quality dimensions more relevant to education, such as difficulty, cognitive level, and diversity. This inevitably introduces subjectivity from the evaluator, affecting the objectivity of the evaluation, which is not conducive to comparison and exchange among similar studies, and also increases research costs and time, affecting research efficiency. Although some studies have designed other automatic evaluation schemes for the cognitive level and diversity of questions, these schemes are still based on semantic similarity evaluation, whereas factors such as the complexity of knowledge, the degree of association between question answers and context have a greater impact on cognitive levels than semantic similarity between questions.

Manual schemes

Automatic evaluation holds significant advantages in terms of research efficiency and cost. However, existing reliable evaluation strategies are largely confined to the realm of semantic similarity assessment. When it comes to evaluating the quality of questions for educational applications, greater emphasis should be placed on dimensions such as educational value embedded in the questions. In this regard, human evaluation by educational experts still plays an irreplaceable role.

Manual evaluation typically covers three main criteria: naturalness (fluency, relevance, and answerability), difficulty, and helpfulness (educational value), which may be expressed differently or include other standards in different studies. The evaluation method usually involves experts, crowd-sourced participants, or the authors themselves, who annotate the generated questions based on specific metrics, and finally, the evaluation results are derived through statistics. For example, in the research (Gao et al., 2019), the difficulty of questions is divided into three levels: 1 (easy), 2 (medium), and 3 (difficult), requiring evaluators to score each question. In the research (Wang et al., 2018), evaluators are asked to mark each question as “true/false” for fluency (coherent and grammatically correct), question-answer relevance, and machine generation. Steuer et al. (2021) invited three trained education experts to evaluate question quality based on the annotation scheme developed by Horbach et al. (2020), which consists of 6 binary indicators, 2 ternary indicators, and 1 quinary indicator, covering various aspects such as understandability, content relevance, naturalness, educational value, and cognitive levels of the questions. Obviously, manual evaluation is time-consuming and labor-intensive, requires high professional knowledge and experience from evaluators, and due to the subjectivity of the evaluation method, it is difficult to unify standards, leading to inconsistent evaluation results and limited widespread application.