Bidirectional matching and aggregation network for few-shot relation extraction

Relation extraction

In recent years, the distant supervision method (Mintz et al., 2009) has been widely used in relation extraction, while achieving excellent result, which also brings the problem of noise in data annotation quality and long-tail distribution of data. For the noise problem, the introduction of the attention mechanism (Luo et al., 2018), the segmental convolutional neural networks (Li et al., 2020) and the multiple instance learning (Lin et al., 2021) have effectively improved the accuracy of automatic annotation. For the problem of long-tailed distribution of data, Zhang et al. (2019) proposed a distant supervision relation extraction method for long-tailed distribution of data, which transfers the knowledge of a large number of identified labeled instances to the relation extraction of scarcity instance classes. Subsequently, Cao et al. (2021) proposed a general method for learning relational prototypes from unlabeled text, which uses relational prototypes to transfer knowledge from head classes to tail classes, therefore improving the performance of relation extraction for tail classes.

Few-shot relation extraction

Early few-shot learning was used in image classification tasks (Wang et al., 2019; Fu, Zhou & Chen, 2022), which aims to classify instances of a query set with a small number of samples by learning generic features of each class. Han et al. (2018) first defined relation extraction as a few-shot learning task and proposed the few-shot relation extraction dataset FewRel, which was generated using distant supervision annotation method and manual annotation method. Gao et al. (2019a) proposed the prototype network of hybrid attention that uses the instance-level attention mechanism to filter out representative support instances and focuses on the important dimensions in the feature space using the feature-level attention mechanism. In the same year, Gao et al. (2019b) proposed the FewRel 2.0 dataset with (N + 1) relation classes, which added cross-domain data and none-of-the-above data to the FewRel dataset. To strengthen the connection between relational prototypes and query sets, Ye & Ling (2019) proposed a multi-level matching and aggregation network that considers the matching between support instances and query instances at the instance level to compute relational prototypes. Ren et al. (2020) proposed the two-stage prototype network for the incremental few-shot relation extraction task, which serves to dynamically identify novel relation in support instances. Liu et al. (2021) proposes a global transformed prototype network based on the learning discriminant paradigm, which extracts relations by learning the differential information between query instances and all target classes. This capability of the model is better suited to the processing of cross-domain data. Wang et al. (2022) proposed the hybrid enhanced prototype network model which enhances the scale and utilization of annotated data.

Task definition

In the few-shot learning method, the dataset is generally divided into a training set and a test set. The training set consists of the support set ( $t_{\mathrm{t}\mathrm{s}}$) and the query set ( $t_{\mathrm{t}\mathrm{q}}$). If there are N relations in the ( $t_{\mathrm{t}\mathrm{s}}$) set, each containing K instances, then this few-shot learning is called the N-way K-shot task, and the label of each ( $t_{\mathrm{t}\mathrm{q}}$) set is one of these N relation labels. The test set is also divided into the support set ( $t_{cs}$) and the query set ( $t_{cq}$) in order to follow the consistency of the test environment with the training environment. The prediction query instance relation in BMAN based on few-shot learning for forward and inverse processes can be expressed as:

(1) $$P(l|S,q)={\displaystyle \frac{\exp (f(\{s_k^l{\}}_{k=1}^K,q))}{{\sum }_{i=1}^N\exp (f(\{s_k^i{\}}_{k=1}^K,q))},}$$

(2) $$\hat{l}=\underset{l}{\arg max}\mathrm{P}(l|S,q),$$

where $S$ represents the forward support set $S=\{s_k^i;i=1,...,N,k=1,...,K\}$, $s_k^i$ is the k-th instance of class $i$, $l$ is the label of the support instance, $\hat{l}$ is the prediction label for the query instance, the function $f(\{s_k{\}}_{k=1}^K,q)$ is used to calculate the degree of match between the query instance $q$ and the support instances $\{s_k^{}{\}}_{k=1}^K$. In the training stage, we used the cross-entropy loss function which reacts to the distance between the predicted label and the true label:

(3) $$J_{forward}=-{\displaystyle \frac{1}{R}\sum _{(q,l)\in Q}^{}\log P{(l|S,q)}_{forward},}$$

(4) $$J_{reverse}=-{\displaystyle \frac{1}{R^{\mathrm{\prime }}}\sum _{(q^{\mathrm{\prime }},l^{\mathrm{\prime }})\in Q^{\mathrm{\prime }}}^{}\log P{(l|S^{\mathrm{\prime }},q^{\mathrm{\prime }})}_{reverse},}$$

where ${\mathrm{J}}_{forward}$ is the forward loss function, $Q$ represents the forward query set $Q=\{(q_j,l_j);j=1,...,R\}$, and R is the sample size of the forward selected query set, $l_j\in \{q,...,N\}$ is the relation label of instance $q_j$. ${\mathrm{J}}_{reverse}$ is the reverse loss function $Q^{\mathrm{\prime }}$ represents the reverse query set $Q^{\mathrm{\prime }}=\{(q_j^{\mathrm{\prime }},l_j);j=1,...,R^{\mathrm{\prime }}\}$, $l_j^{\mathrm{\prime }}\in \{q,...,N\}$ is the relational label of instance $q_j^{\mathrm{\prime }}$. Combine Eqs. (3) and (4) to form the final function:

(5) $${\mathrm{J}}_{total}={\mathrm{J}}_{forward}+{\displaystyle \frac{\alpha }{x}{\mathrm{J}}_{reverse},}$$where $\alpha$ is the weighting parameter, $x$ is the group number of the query set in the reverse mechanism.

In summary, the BMAN model based on few-shot learning generates N-way K-shot tasks by randomly selecting instances in the dataset, and utilizes the N-way K-shot tasks to learn features of relation classes. Next, query instances are matched with the calculated relation class features, and the relations between entities in the query instances are predicted by the matching degree.

Bidirectional matching and aggregation network framework

The article proposes the bidirectional matching and aggregation network based on MLMAN. The model BMAN has four main components and the framework is shown in Fig. 1.

1. Instances encoder. Bidirectional Encoder Representations from Transformer (BERT) (Devlin et al., 2018) extracts the features of the instances based on different perspectives and embeds the instance features into a vector representation.

2. Multi-level matching and aggregation. The results of the sentence encoder are fed into MLMAN to obtain a matching representation of the support instance and the query instance, then the match between the support instance and the query instance is calculated.

3. Relational prototype. The matching results obtained in the previous step are used as weights for the aggregated support instances to obtain a vector representation of the relation class.

4. Bidirectional matching. The results of the forward matching are used as the support set for the reverse matching, while the results of the reverse matching guide the forward matching, which makes the obtained relational prototypes more accurate.

Data enhancement method

The conversion of forward and reverse data in the framework utilizes data enhancement method, which is responsible for generating the reverse support set ${\mathrm{S}}^{\mathrm{\prime }}$ by the forward query set $\mathrm{Q}$ and generating the reverse query set ${\mathrm{Q}}^{\mathrm{\prime }}$. The design of this method is a simple to complex process. The reverse support set ${\mathrm{S}}^{\mathrm{\prime }}$ and the reverse query set ${\mathrm{Q}}^{\mathrm{\prime }}$ are divided directly on the single direction queue of instance sets, which is called BMAN-simple, as shown in Fig. 2. But such an approach does not exploit the results of forward matching and severs the connection between forward and reverse in the matching and aggregation network, which is not the result we expected. To address the above shortcoming, we use the forward matching results to recombine the instances in the query set $\mathrm{Q}$ to obtain multiple reverse support sets ${\mathrm{S}}^{\mathrm{\prime }}$, as shown in Fig. 2, and randomly select instances of the same class as the support set S to form the reverse query set ${\mathrm{Q}}^{\mathrm{\prime }}$, which is called BMAN-link. We also found that the labels of the m consecutive instances in the k-shot task were identical, so that the logic of BMAN-link was not applicable to the k-shot task. This article also improves on BMAN-link to form the final BMAN model, which uses the 1-shot task as a special k-shot, as shown in Fig. 2. In order to ensure that the reverse support set ${\mathrm{S}}^{\mathrm{\prime }}$ contains instances of n relation classes, this article devises solutions for two possible scenarios:

1. Reuse instances within instance-poor classes when the distribution of forward matching results is unbalanced.

2. When instances of a certain class are completely missing, instances of this class in the forward support set S are introduced.

This method maintains the range of relation classes while expanding the size of the instances, which allows the model to obtain more information about the relational prototypes while avoiding overfitting problem. The specific code implementation is shown in Algorithm 1.

Dataset

This article evaluates the BMAN model on two publicly available datasets, details of which are shown in Table 2. The dataset FewRel (Han et al., 2018) is the first proposed dataset for few-shot relation extraction, which is based on Wikipedia text and constructed using distant supervised learning and manual annotation, with the data distribution obeying a balanced distribution. The dataset FewRel contains 100 relations with 700 instances each, where the train set contains 64 relation classes, the validation set contains 16 relation classes and the test set contains 20 relation classes. The dataset FewRel 2.0 (Gao et al., 2019b) adds cross-domain data based on dataset FewRel which introduces biomedical texts from the PubMed dataset and UMLS dataset to generate the test set, where 25 relation classes with 100 instances each are included, and 17 relations from the SemEval dataset are selected to construct the validation set. The datasets FewRel 2.0 and FewRel share the same train set.

Local verification

Hyperparameter

The effect of hyperparameters in the model was experimentally investigated, with the model trained on a device with an NVIDIA RTX3090 24G GPU. Due to GPU video memory limitations, the experiments were set up with a query set sample of $R=5\cdot n$, where $n$ is the number of total relation classes, and the value in the learning rate $learn\mathrm{\_}rate\in (1e-4,1e-3,1e-2,2e-2,1\mathrm{e}-1,2e-1,3e-1)$ that makes the model accurate is determined, as shown in Fig. 3. As can be seen from the figure, on the FewRel, FewRel 2.0 dataset. The model BMAN obtained the highest accuracy when the learning rate was 0.2.

In this article, we analyzed the effect of the number of instances and relations on model performance at the dataset level and task level respectively. We set the number of instances of each relation in the train set to 700, 500, 300 and 100, respectively, which generated four training sets of train_700, train_500, train_300 and train_100. The BMAN model was trained on these train sets, and the results are shown in Fig. 4. The results show that the accuracy of the 5-way 1-shot task fluctuates within 1.15 on the FewRel dataset and within 1.24 on the FewRel 2.0 dataset. The accuracy of the 10-way 1-shot task fluctuates within 1.61 on the FewRel dataset and within 1.58 on the FewRel 2.0 dataset. The accuracy of the 5-way 5-shot task fluctuates within 0.72 on the FewRel dataset and within 1.21 on the FewRel 2.0 dataset. The accuracy of the 10-way 5-shot task fluctuates within 0.53 on the FewRel dataset and within 1.19 on the FewRel 2.0 dataset. We set the number of relations in the train set to 64, 44 and 24, respectively, which generated three train sets of train_64, train_44 and train_24 for verifying the effect of the number of relations on BMAN, and the results are shown in Fig. 5. The results show that the accuracy of the five-way one-shot task fluctuates within 7.5 on the FewRel dataset and within 3.62 on the FewRel 2.0 dataset. The accuracy of the 10-way one-shot task fluctuates within 8.16 on the FewRel dataset and within 1.64 on the FewRel 2.0 dataset. The accuracy of the five-way five-shot task fluctuates within 2.69 on the FewRel dataset and within 5.27 on the FewRel 2.0 dataset. The accuracy of the 10-way five-shot task fluctuates within 5.11on the FewRel dataset and within 6.74 on the FewRel 2.0 dataset. The results of the above two experiments show that the number of instances have a weak effect on BMAN performance and the number of relations have a strong effect on BMAN performance at the dataset level, which indicates that BMAN can be used in scenarios with sufficient relations and few instances, and can effectively solve the problem of lack of tail instances in the long-tail distribution of data. The experimental results also show that the performance of the BMAN model decreases with increasing ‘way’ and increases with increasing ‘shot’ at the task level, which means that the performance of the model decreases as the number of relations in the task increases and increases as the number of instances selected for the task increases.

To directly validate the performance of the model on the long-tailed distribution datasets, we constructed the long-tailed distribution datasets FewRel_L and FewRel 2.0_L based on random sampling of the original datasets, where the relation classes of the train, validation and test sets were kept constant and the number of instances was adjusted, as shown in Fig. 6. The datasets FewRel_L and FewRel 2.0_L share the same training set and the validation results of the BMAN model are shown in Table 4. The accuracy of the BMAN model on the datasets FewRel_L for the tasks five-way one-shot, 10-way one-shot, five-way five-shot and 10-way five-shot lost 0.15, 0.94, 0.84 and 0.38 respectively. The accuracy of the BMAN model lost 0.23, 1.02, 1.13, and 0.39 on the datasets FewRel 2.0_L for the tasks five-way one-shot, 10-way one-shot, five-way five-shot, and 10-way five-shot respectively. The results show that the model BMAN keeps excellent performance on the long-tailed distribution dataset, which indicates that the model BMAN is effective for solving the long-tailed distribution of the data.

Since both the BMAN and the MLMAN use matching and aggregation network, this article evaluates the time complexity of the BMAN for different task scenarios using MLMAN as the baseline, as shown in Fig. 7.

The comparison in the Fig. 7 shows that the accuracy of the BMAN model improves which comes at the cost of time complexity. In the datasets FewRel and FewRel2.0, the accuracy improvements in the five-way one-shot task were 14.63% and 13.89% respectively, while the corresponding time complexity improvements were 490.95 and 478.04% respectively. The accuracy improvements in the 10-way one-shot task were 22.20% and 29.60% respectively, while the corresponding time complexity improvements were 477.30% and 473.44% respectively. The accuracy improvements in the five-way five-shot task were 5.95% and 22.81% respectively, while the corresponding time complexity improvements were 444.55% and 441.46% respectively. The accuracy improvements in the 10-way five-shot task were 9.04% and 34.59% respectively, while the corresponding time complexity improvements were 427.02% and 449.57% respectively. The increase in time complexity is due to the fact that a set of forward query set can be mapped to multiple sets of reverse support set during the bidirectional training process with increased computational effort for reverse matching.

This article also selects methods as a baseline for comparison and evaluates the performance of the BMAN model for relation extraction in vertical domain. The evaluation metrics in this article follow the settings in FewRel and include accuracy on the five-way one-shot, five-way five-shot, 10-way one-shot and 10-way five-shot tasks. This article compares the BMAN with the following baselines:

• MLMAN (Ye & Ling, 2019): This approach interactively encodes support instances with query instances, and the matching process strengthens the connection between the query set and the relational prototype.

• CTEG (Wang et al., 2020): The model uses entity-guided attention mechanisms and confusion-aware training to distinguish between problems of relational confusion.

• TPN(BERT) (Wen et al., 2021): The method integrates the transformer model into the prototype network and uses the pre-trained BERT as the encoder for the model.

• ConceptFERE (Yang et al., 2021): The model enhances the influence of entity concepts on relations in relation extraction.

• COL Final (Ding et al., 2021): The method learns the relational prototype from contextual information, which focuses on the influence of semantics on the relational prototype, and uses spherical coordinates as the basis for prototype interpretation.

• GTPN (Liu et al., 2021): The model is based on the learning-to-discriminate paradigm and focuses more on the discriminatory knowledge between all candidate classes.

As can be seen from Table 5, BMAN achieved the highest accuracy on the FewRel dataset compared to previous studies. BMAN compared to COL Fina respectively improved by 2.1% and 5.71% on the five-way one-shot and 10-way one-shot tasks. BMAN compared to GTPN respectively improved by 0.24% and by 0.46% on the five-way five-shot and 10-way five-shot tasks. It is proven that the bidirectional matching and aggregation mechanism can effectively improve the accuracy of relation extraction.

• Proto-Bert (Gao et al., 2019b): Bert is utilized as an encoder in the prototype network.

• Proto-CNN (Gao et al., 2019b): CNN is utilized as an encoder in the prototype network.

• Proto-Adv (Bert) (Gao et al., 2019b): Bert is utilized as an encoder in the prototype network for adversarial training.

• Proto-Adv (CNN) (Gao et al., 2019b): CNN is utilized as an encoder in the prototype network for adversarial training.

Table 6 shows that BMAN achieves the best performance in these tasks compared to other approaches based on the learning-to-matching paradigm, but with a significant difference compared to GTPN. The reason for this result is that the BMAN model focuses more on matching between relational prototype features and query instance features, whereas GTPN focuses more on discriminating between different features of query instances and target relation classes, which is more important for relation extraction of cross-domain data. Specifically, the main advantage of GTPN is that all candidate relation classes are considered jointly rather than independently when extracting relations, which allows the model to more accurately use discriminative information between classes to distinguish between relation confusions caused by cross-domains. In contrast, the BMAN model has a larger error in measuring the degree of similarity between query instances and out-of-domain relations in the cross-domain case. However, the measurement of query instance-prototype similarity directly affects the accuracy of relation extraction within the domain, and BMAN’s bidirectional matching and aggregation enhances the measurement of query instance-prototype similarity. Thus, we proposed the BMAN model that outperformed the GTPN model in the dataset FewRel, while the GTPN model outperformed the BMAN model in the dataset FewRel 2.0.