Beyond top-k: knowledge reasoning for multi-answer temporal questions based on revalidation framework

Boot node representation

In order to use the answer information to guide the question reasoning, the question q and the candidate answer set $\mathcal{A}$ provided by other question answering schemes are together inserted into the knowledge graph as a special node, known as boot node (boot), denoted as [q; a], as shown in Fig. 3. Herein, $\mathcal{A}$ can be a traditional form of Top-k solution to question q given by any question answering scheme, and the standard answer in the candidate solution set $\mathcal{A}$ is clearly marked. In the special nodes formed by Q&A pairs, the question is taken as the starting point of the reasoning model, and the answer as the end point, implicitly expressing the information of the question and answer context. The boot node is associated with entities contained in the question, and the mapping item of the boot node and the marked standard answer node in the knowledge graph is linked, and the new relation “gold answer” is given, which is shown by the orange dotted line in Fig. 3. Therefore, a new answer-guided knowledge graph is constructed between the boot node and the knowledge graph, and between the answer node and the corresponding boot node, known as inference graph G_R herein.

The boot node is regarded as a long sequence text and encoded by BERT, where f_e is the encoding function. (1)${\displaystyle {\mathit{boot}}^{BERT}=f_e\left(text\left(boot\right)\right).}$

After the boot node is given, the subgraph $G_{sub}^{boot}=\left(v_{sub}^{boot},e_{sub}^{boot}\right)$ after entity link is extracted from knowledge graph G = (V, E), where V is the set of entity node of the knowledge graph, E is the set of relationships between entity nodes, $v_{sub}^{boot}$ is the entity nodes in all boot nodes extracted from the graph, $e_{sub}^{boot}$ is the relationship nodes in all the boot nodes extracted from the graph, and $G_{sub}^{boot}$ is the subgraph associated with the boot node extracted from the knowledge graph.

Node-question relevance scoring

There are many paths unrelated to the question in the subgraph after entity link disambiguation. As shown in Fig. 1, Martin Van Buren’s path as president is unrelated to his path as Secretary of State. These unrelated paths cause the model to waste a lot of time in the inference process to exclude invalid paths. To address this problem, this paper uses the question correlation fact determination module to calculate the similarity score between the boot node and KG fact node. (2)${\displaystyle {\mathit{S}}_{sub}^{boot}=f_h\left(f_e\left(\left[text\left(boot\right);text\left(v_{sub}^{boot}\right)\right]\right)\right),}$ where f_h is a function to obtain the head of the language model (here it is used to obtain the head of BERT), and f_h(f_e()) is the probability that the boot node is connected to the subgraph node; ${\mathit{S}}_{sub}^{boot}$ is the score of correlation between the boot node and the subgraph node, which describes the importance of each node to the boot node, and is used to prune the inference graph G_R.

Initial answer determination

The answer with the highest score in the question answering system has the greatest probability of being the standard answer. This paper therefore finds out the most likely answer to the multi-answer question through subgraph reasoning, and regards it as the correct answer. MATQA’s reasoning process is based on the graph attention GAT framework.

In an l-layer graph network model, for a node v ∈ V_sub in any subgraph, vector initialization is performed by BERT encoding, i.e., ${\mathit{h}}_v^0=f_e\left(text\left(v\right)\right)$. Then the updating model can be expressed as: (3)${\displaystyle {\mathit{h}}_v^{l+1}=\left(\sum _{n\in N_v\cup \left\{v\right\}}{\delta }_{nv}{\mathit{m}}_{nv}\right)+{\mathit{h}}_v^l,}$ where ${\mathit{h}}_v^{l+1}\in {\mathbb{R}}^D$ is the representation of node v ∈ V (in the form of a D-dimensional vector), N_v is the set of neighbors of node v, m_nv is the message from each neighbor node n to node v, and δ_nv is the weight of the message from node n to node v. The calculation of message m_nv should take into account the characteristic ${\mathit{h}}_n^l$, type u_n, and time attribute t_n of the node, as well as the embedded relation r_nv. The calculation formula is as follows: (4)${\displaystyle {\mathit{m}}_{nv}=Linear\left({\mathit{h}}_n^l,{\mathit{u}}_n,{\mathit{t}}_n,{\mathit{r}}_{nv}\right),}$ where u_n is the type’s one-hot encoding of the neighbor n of node v, t_n is the embedded time attribute of neighbor node n, and r_nv is the embedded relation between nodes n to v.

To calculate the attention weight vector of nodes n to v, query vector q and key vector k are constructed according to node types: (5)${\displaystyle \begin{array}{c}{\mathit{q}}_n=Linear\left({\mathit{h}}_n^l,{\mathit{u}}_n,{\mathit{S}}_n^{boot}\right)\hfill \\ {\mathit{k}}_v=Linear\left({\mathit{h}}_v^l,{\mathit{u}}_v,{\mathit{S}}_v^{boot},{\mathit{r}}_{nv}\right)\hfill \end{array},}$ where Linear is a linear transformation that converts the input into a D-dimensional vector. ${\mathit{S}}_n^{boot}$ and ${\mathit{S}}_v^{boot}$ is the correlation score between the boot node and nodes n and v. The final attention weight vector can be obtained by formula (Eq. 6) below. (6)${\displaystyle {\delta }_{nv}=\frac{exp\left({\gamma }_{nv}\right)}{\sum _{v^{\prime }\in N_n\cup \left\{n\right\}}exp\left({\gamma }_{n{v}^{\prime }}\right)},{\gamma }_{nv}=\frac{{\mathit{q}}_n^T{\mathit{k}}_v}{\sqrt{D}}.}$

Then the reasoning process of the initial answer $p\left(a_0^i|q\right)$ is given by: (7)${\displaystyle p\left(a_0|q_i\right)=exp\left(MLP\left({\mathit{boot}}^{BERT},{\mathit{h}}_{boot}^l,G_{sub}^{pooling}\right)\right),}$ where boot^BERT is the vector representation of boot node, ${\mathit{h}}_{boot}^l$ is the updating representation of the boot node at the l-th layer, and $G_{sub}^{pooling}$ is the pooling representation of subgraph.

Answer clustering

After the initial answer a₀ is obtained, the rest of the answers to the question should be deduced. Since all answers to the question should meet the same constraints, including the semantic and time constraints, MATQA processes the other answers through clustering. In order to correctly measure the gap between the alternative answer and the initial answer a₀, the subgraph path (V_sub, E_sub, a_other) of the alternative answer is extracted to calculate the semantic similarity score between it and the path (V_sub, E_sub, a₀) of the initial answer. (8)${\displaystyle {\mathit{S}}_{\text{semantic}}=cos\left[\left(V_{\text{sub}},E_{\text{sub}},a_{\text{other}}\right),\left(V_{\text{sub}},E_{\text{sub}},a_0\right)\right].}$

The final answer to each question is constrained by the time interval. Therefore, the matching between the time interval of the fact and the real time interval of the question can exclude the answer that does not satisfy the condition. KG retrieval and TimeML (Pustejovsky et al., 2003) are used to calculate the time constraint interval of the question, which is [T_s, T_e] (T_s and T_e are the start time and end time, respectively). At the same time, the time interval [$T_s^{other},T_e^{other}$] of the fact corresponding to the alternative answer is extracted. The final predicted score of time similarity S_time can be obtained by: (9)${\displaystyle {\mathit{S}}_{\text{time}}=ReLU\left\{\begin{array}{c}1,T_s<T_s^{\text{other}}\text{and}{T}_e^{other}<T_e\hfill \\ -1,T_s^{\text{other}}<T_s\text{or}{T}_e^{\text{other}}>T_e\hfill \end{array}\right\}.}$

We use the K-means algorithm with K = 2 for clustering, where one cluster center is set to the initial answer a₀. Setting K to 2 is because we believe that the correct answers will be closely distributed in the neighborhood of the top-1 answers in the initial answer list in the entire answer candidate solution space. Therefore, when the number of clusters is set to 2, the correct answer combinations can be aggregated into one cluster, while the remaining answers will be classified into another cluster.

The ReLU function is commonly used as an activation function, but this article uses its rectifying properties to filter S_time: when the answer does not meet the time constraints, the score is truncated to 0 through ReLU. The answers that satisfy the semantic and time constraints after clustering are regarded as the true predicted answers $\mathcal{A}$ to the question q. Each row of Top-k is a combination of answers, as shown in the expected answer expressions in Fig. 1.

Datasets

TimeQuestions (Jia et al., 2021) is a wikidata-based question-answering data set consisting of 16,181 Q&A pairs, among which 9,708 questions are used for training, 3,236 for verification and 3,237 for testing. The type of each question (explicit, implicit, time, and order) is indicated in the Q&A pairs. At the same time, the signal words for time interaction in the question are specified, such as before/after, start/end, etc. In order to process the multi-answer questions, all question pairs with more than one answer are extracted from the TimeQuestions data set to construct the multi-answer TimeQuestions data set. The new multi-answer question dataset contains 2,264 training sets, 778 verification sets and 801 test sets, and the labels of the question type and time signal.

Evaluation metrics

Two measures are used to evaluate the quality of answers to the multi-answer question.

• P@1^m (the precision of multi-answers): For a new answer form given in a question, the highest-ranked combination of answers has a precision of 1 when the combination is exactly the same as the standard answers (both in the quantity and the label), which is denoted as ${P@1}_{hard}^m$. When the highest-ranked answer combination contains all the standard answers, that is, the first result of the prediction includes other results besides the standard answers, it is denoted as ${P@1}_{soft}^m$ with broader constraints.

• Hits@5^m (the hits of multi-answers): The combination of answers depends on the number and label of answers. The label needs to satisfy the semantic matching relation of the question, and the number is all possible solutions that satisfy the semantic constraints. Because of the complexity of language questions, semantic constraints cannot be fully satisfied, and there are many possible combinations of answers. Under the new answer expression form, the first five groups of answers are ranked in descending order of the proportion of the standard answers on the list. If a list containing any subset of the standard answer appears in the first five positions, it is set to 1, otherwise to 0.

Baselines

The goal of the traditional Top-k based QA system on multi-answer questions is to predict every answer that may belong to the correct answer combination, while MATQA, as a plug-in component of the QA system, converts the goal of the QA system into directly predicting answer combinations. The system’s answer output has been completely changed so that the prediction results are presented in the form of a list of answer combinations. For this reason, the experiment in this section aims to reflect the effectiveness of this method on multi-answer questions through the metrics P@1^m and Hits@5^m designed for this predicted answer form, rather than verifying the superiority of this method compared to other methods.

• TransE: it is the most classical vector embedding method which completes the missing answers according to the translational semantic invariance law.

• EXAQT (Jia et al., 2021): it is an end-to-end temporal question answering scheme, which for the first time builds the temporal question answering system on wikidata, a large-scale open-domain knowledge graph. It does not require the process of constructing a temporal knowledge graph. The final answer prediction and accuracy is performed using a relational graph convolution network (R-GCN) by augmenting the embedding of subgraphs and questions, performing temporal augmentation of subgraphs, or reconstructing subgraphs to augment recall in three ways.

• TERQA (Yao et al., 2022): On the basis of EXAQT, inspired by capsule network, TERQA improved the fusion of time features and triplet features and learned the exact dependence between time features and triplet facts, which enhanced the accuracy of the model to predict the answer.

Experimental settings

MATQA uses PyTorch for implementation, and sets the vector embedding dimension after BERT initialization to 200. It has five layers of GNN, each of which with a dropout of 0.2. Moreover, it uses Adam for initial answer inference optimization and ReLU as a filter on time constraint scores. Furthermore, batch_size is set to 32, learning rate to 2e−3, and cluster number to 2.

Key findings

Table 1 shows the effects of multi-answer judgment on the multi-answer question data set. The index ${P@1}_{hard}^m$ demonstrates that MATQA can improve the traditional Top-k expression form to make each line a new form of a list of answers, which is consistent with the expected human expression form in Fig. 1. Therefore, MATQA can better meet user’s requirements on the number and accuracy of questions with multiple answers. At the same time, MATQA has proved that its effectiveness is largely related to the alternative answers provided. That is, the more accurate the candidate answers, the more accurate the initial answer, and the better the final result after clustering.

Through the revalidation framework of “initial answer → clustering”, MATQA can provide a solution to the multi-answer temporal reasoning question. The primary shortcoming of MATQA is that its final output is largely affected by the initial result. In other words, in the case of an incorrect initial answer, the subsequent clustering module cannot correct it and can only make invalid predictions on a wrong basis.

Disambiguation experiment

Table 2 shows the results of MATQA after removing each module. It can be seen that the introduction of the boot node enables the question and the candidate answers to inspire the inference model. In addition, the boot nodes have positive feedback to P@1^m. In the case of no boot nodes, the P@1^m score is the lowest relative to the case with a boot node, which means the QA model cannot get the information guidance of hidden answer, and the Q&A context cannot be updated with KG, which cannot bridge the information gap between question and knowledge graph and thus damages the system performance (${P@1}_{hard}^m$:40.2% → 38.2%, ${P@1}_{soft}^m$:43.9% → 39.1%).

When semantic constraints are removed during clustering, the model effect declines most seriously, because the clustering of answers mainly measures the degree of fact similarity. Additionally, among temporal questions, a large proportion have answers within a specific time constraint interval. When time constraint is removed, the entities of the answers cannot be measured by time constraint, which will easily lead to incorrect answers. Finally, the addition of the boot node makes up the information gap between the question context and the knowledge graph, and has a great influence on the determination of the initial answer. Removing modules from GNN also has an effect on the prediction of the final initial answer.

Typical questions

The effectiveness of MATQA is fully demonstrated by three typical questions. In Table 3, the question Q1 has the standard answers of “Super Bowl ‘IX’, ‘X’, ‘XIII’, ‘XIV’, ‘XL’, ‘XLIII”’. The model has accurately predicted the number of answers and the correct answer. In the traditional Top-K method, it is difficult to obtain the correct answer combination due to the difficult K setting. For example, when the setting of k is less than the number of correct answers, it will result in the output of incomplete answers. Taking Q1 in Table 3 as an example, when k = 3, the three answers with the highest scores will be output, namely: IX, X, XIII , the correct answer with the lower score is lost. For another example, in the answer list obtained based on the top-k method, there may be cases where the correct answer is not the highest-scoring answer. Even if the correct k value is selected, it may lead to the wrong selection of the final combined answer. It is proved that MATQA framework has a good effect on the processing of multi-answer temporal questions, and makes up the defects of traditional top-k which cannot show the number of answers and has false positive results.

Error types

As shown in Table 4, we selected two questions Q1 and Q2 with numerical answer types as cases of incorrect answers for analysis. question Q1 expected a numeric answer of 2.7, but instead returned multiple unrelated entities as the answer. This shows that MATQA still cannot accurately determine the number of answers through semantic and time constraints for some single-answer questions, and there is room for further improvement. question Q2 expected to get two numerical answers “1995” and “1998”, but actually got a single entity as the answer. We believe that this phenomenon may be related to the initial answer generation of the upstream task. As we describe in Section: Initial result determination, MATQA will use GNN to perform inference on the extracted subgraph to obtain a preliminary answer. The effect of inference on the subgraph depends on the extent to which GNN can accurately learn the nodes. feature. For entity-type answers, each entity can have multiple neighbors. The rich neighborhood structure allows GNN to capture the characteristics of entities very well. However, for numerical nodes, most of them are only used for directly related nodes. For example, the expected answer of Q2 is “2.7”. This value is quite special and it is difficult to find a second node that refers to this value. Therefore, the GNN is likely to make errors in capturing its features, which in turn leads to the wrong exclusion of the answer, so the downstream The clustering module will not be able to get the correct answer.