Effective sentence-level relation extraction model using entity-centric dependency tree

Task definition

In this article, we focus on sentence-level relation extraction. Let T = {w₀, w₁, ..., w_|T|} denote the input words in a sentence and e_sbj, e_obj ∈ T denote the subject and object entity spans. When given a predefined set of relation categories denoted as R, the objective of relation extraction is to predict the relation r ∈ R between entities within the sentence.

Entity-centric dependency tree

Syntactic information of a dependency tree is an essential feature for relation extraction. However, as observed in Fig. 1, hard pruning centered on the subtree between entities can result in the loss of crucial lexical information. To address this problem, we reconstruct dependency trees in an entity-centric manner. Since there are two types of entities in a sentence (i.e., subject and object entities), we create a separate dependency tree for each entity. Algorithm 1  describes the process of constructing an entity-centric dependency tree.

 
_______________________ 
Algorithm 1 Build Entity-centric Dependency Tree__________________________________ 
Input: words T = {w0,w2,...,wt} 
    entity span e = {we1,we2,...,we|e|} 
    dependency tree D 
Output: Entity-centric dependency tree De 
  1:  Initialize entity-centric dependency tree with De ←{root root − − − → e} 
 2:  for i ← 0 to |T| do 
 3:       if wi / ∈ e then 
 4:            Initialize Ke with empty set 
  5:            for wj ∈ e do 
 6:                  Ke ← Ke ∪ distance(wi,wj) 
  7:            end for 
 8:            k ← min(Ke) 
  9:            De ← De ∪{e con:k − − − → w 
i} 
10:       end if 
11:  end for 
12:  return De______________________________________________________________________________________    

In Algorithm 1 , T denotes a set of all words in a sentence, e denotes the current entity span, and D denotes a dependency tree of the sentence. The outputs D_e denote the entity-centric tree reconstructed from the original dependency tree D. First, we set the entity as the root of the entity-centric dependency tree (line 1). Next, we traverse all words except for the entity and calculate a syntactic distance between the words and the entity (lines 2-11). Since the entity can consist of multiple words, we calculate the distance between the current word w_i and all words comprising the entity (lines 5-7). K_e represents temporary lists for storing the distance between words comprising the entity and w_i. The function distance(w_i, w_j) denotes the distance in the original dependency tree D between w_i and w_j. For example, in the second sentence of Fig. 1, the entity ‘Audran’ can reach the word ‘wife’ through two hops in the dependency tree. After calculating the distance to all words comprising the entity, we consider the minimum value as the distance between the current word w_i and the entity (line 8). This distance determines the labels of each word with respect to the entity in the entity-centric dependency tree (line 9). Figure 2 illustrates how a general dependency tree is reconstructed as an entity-centric dependency tree. In the new tree, dependency relations are replaced with syntactic distances. Next, the entity-centric dependency tree is inputted to the graph attention network, and entity-centric lexical information is extracted by soft pruning.

Relation extraction model

Figure 3 illustrates the overall architecture of the relation extraction model based on an entity-centric dependency tree. Most recent studies in natural language processing have used transformer-based pre-trained language models to extract contextual knowledge from given sentences or documents. We encoded each word in the sentences using transformer-based pre-trained language models. Once each token has been encoded by the language model, the average of the token encoding vectors corresponding to the entity is used as the entity encoding vector. For example, when k and l denote the start and end indices of the entity, the entity encoding vector h^e is as follows: (1)${\displaystyle h^e=\frac{1}{l-k+1}\sum _{i=k}^l{h}_i}$ where h_i is i-th word encoding vector obtained from pre-trained language model.

Edges and syntactic distances in an entity-centric dependency tree are input into two separate networks. Edges provide entity-oriented lexical information in a graph attention network, while syntactic distances provide syntactic information between two entities. Entity-centric dependency trees connect every word node to the root node (entity). Therefore, in the graph attention network, attention between the entity and the word encoding vectors is used to perform soft pruning. The graph attention network is as follows: (2)${\displaystyle g^e=\sum _{i\in {\mathbb{N}}_e}{\alpha }_i^e{h}_i}$ (3)${\displaystyle {\alpha }_i^e=\frac{exp\left(s_i^e\right)}{\sum _{j\in {\mathbb{N}}_e}exp\left(s_j^e\right)}}$ (4)${\displaystyle s_i^e=\frac{h^e\cdot W_{\alpha }\cdot h_i}{\sqrt{d}}}$ where g^e is the result of entering a dependency tree into a graph attention network, with entity e as the root. Here, ℕe denotes the adjacent nodes (i.e., all tokens except for the entity) of entity e, and ${\alpha }_i^e$ represents the attention weight between the entity e and its i-th adjacent node. The attention score s^e is calculated with bilinear attention, as expressed in Eq. (4), where W_α ∈ ℝ^d×d is the trainable bilinear weight matrix. The attention score s^e is normalized by the hidden size d. Lexical information for relation extraction is obtained in this process.

The syntactic distance is embedded with random initialization. These embeddings are passed through another attention layer, yielding syntactic information between two entities. The attention layer can be expressed as follows: (5)${\displaystyle u=\sum _{t=0}^T{\beta }_t{h}_t}$ (6)${\displaystyle {\beta }_i=\frac{{\beta }_i^{sbj}\times {\beta }_i^{obj}}{\sum _{t=0}^T\left({\beta }_t^{sbj}\times {\beta }_t^{obj}\right)}}$ (7)${\displaystyle {\beta }_i^e=\frac{exp\left(s_i^e\right)}{\sum _{t=0}^{T}exp\left(s_t^e\right)}}$ (8)${\displaystyle s_i^e=\sigma \left(z_i^e{W}_{\beta }+b_{\beta }\right)}$ where $s_i^e$ is calculated using $z_i^e$ as input to a feed-forward neural network (FNN) layer. $z_i^e$ is the embedding vector for the syntactic distance between entity e and the ith word. W_β ∈ ℝ^d_z×1 is a trainable parameter, and d_z is the dimension of the syntactic distance embedding. σ is an activation function such as ReLU (Agarap, 2018). Entity-centric dependency trees are created for both subject and object entities. Equation (6) illustrates combining the attention weights of both trees. ${\beta }_i^{sbj}$ and ${\beta }_i^{obj}$ denote the syntactic attention weights of the subject and object entities, respectively. The co-attention weight β_i reflects common syntactic information between entities. u is the distance attention output vector calculated from β. Through this process, we acquire syntactic information between the entities.

The final entity encoding vector and relation prediction are expressed as follows: (9)${\displaystyle o^e=\sigma \left(W_e\left[h^e;g^e\right]+b_e\right)}$ (10)${\displaystyle r_p=\sigma \left(W_r\left[o^{sbj};o^{obj};u\right]+b_r\right)}$ where o_e is the encoding vector of an entity and is calculated by FNNs with concatenations of entity embedding h^e and graph attention network output g^e. The relation prediction r_p is calculated by FNNs with concatenations of subject encoding o^sbj, object encoding o^obj, and common syntactic information u. Here, W_e ∈ ℝ^2d×d and W_r ∈ ℝ^3d×|R| are a trainable weight matrix. The |R| represents the size of the set of relations. In relation extraction, predicting negative labels (i.e., entity pairs without any relation) is equally crucial as predicting positive labels (i.e., entity pairs with relation). To effectively differentiate them, we train the proposed method using the adapted thresholding loss (ATL) of Zhou et al. (2021). The ATL is as follows: (11)${\displaystyle L_p=-\sum _{r_p\in P_T}\frac{exp\left(logi{t}_{r_p}\right)}{\sum _{r^{\prime }\in P_T\cup \left\{TH\right\}}exp\left(logi{t}_{r^{\prime }}\right)}}$ (12)${\displaystyle L_n=-log\left(\frac{exp\left(logi{t}_{TH}\right)}{\sum _{r^{\prime }\in N_T\cup \left\{TH\right\}}exp\left(logi{t}_{r^{\prime }}\right)}\right)}$ (13)${\displaystyle L=L_p+L_n}$ where L_p is a loss function between positive label logit and a threshold logit, and L_n is a loss function between negative label logit and a threshold logit. Through L_p and L_n, the threshold value is trained to be lower than the logit values of positive labels and higher than the logit values of negative labels.

Dataset and evaluation measure

To evaluate the proposed method, we select TACRED (Zhang et al., 2017), Re-TACRED (Stoica, Platanios & Póczos, 2021) and SemEval 2010 Task 8 dataset (Hendrickx et al., 2010).

TACRED: TACRED is a popular large annotated dataset for sentence-level relation extraction, used in the Text Analysis Conference Knowledge Base Population challenge (TAC-KBP). TACRED covers 42 relation types (including the ‘no_relation’ class) and 16 entity types. The dataset is split into train, development, and evaluation sets, with 68,057, 22,630, and 15,509 sentences, respectively. In total, 78.4% of the data consists of negative samples. The TACRED dataset includes dependency trees analyzed with the Stanford parser. Our method creates entity-centric dependency trees from the given dependency tree.

Re-TACRED: While TACRED is a dataset constructed through human annotation, it includes approximately 25% incorrect labels. Re-TACRED is a significantly improved version of TACRED, involving the pruning of poorly annotated sentences and the resolution of ambiguities in TACRED relation definitions. Re-TACRED encompasses a total of 40 relation types. The dataset includes 58,465 sentences for training, 19,584 sentences for development, and 13,418 sentences for evaluation.

SemEval 2010 Task 8 dataset: Similar to TACRED, the SemEval 2010 Task 8 dataset is designed for sentence-level relation extraction. It is a publicly available relation extraction dataset with less data compared to TACRED. The training and test sets consist of 8,000 sentences and 2,717 sentences, respectively. There are a total of 19 relation types to classify, including the “other” class. Unlike TACRED, dependency trees are not provided. Therefore, we leverage the Stanford parser (https://nlp.stanford.edu/software/nndep.html) for dependency analysis and utilize the information to create entity-centric dependency trees.

The Stanford parser achieves an unlabeled attachment score (UAS) of 91.7% and a labeled attachment score (LAS) of 89.5% on the Penn Treebank dataset (Marcus, Marcinkiewicz & Santorini, 1993). Therefore, it is important to note that the experimental results may contain errors introduced by the Stanford parser. Table 1 describes statistics about TACRED, Re-TACRED and SemEval 2010 Task 8 dataset.

To evaluate the experimental results, we adopt the standard micro precision (P), recall (R), and F1-score (F1). All experimental results are obtained by averaging the results over five independent trials. ${\displaystyle P=\frac{\#ofcorrectpredict}{\#ofalltripleinthemodelpredict}}$ (14)${\displaystyle R=\frac{\#ofcorrectpredict}{\#ofalltripleinthedataset}}$ ${\displaystyle F1=\frac{2PR}{P+R}.}$

Implementation details

We conduct experiments with three popular pre-trained language models: BERT (Devlin et al., 2019), ELECTRA (Clark et al., 2019), and RoBERTa (Liu et al., 2019). The base-scaled model consists of 12 transformer layers, with a hidden size of 768 and 110M trainable parameters. The large-scaled model consists of 24 transformer layers, with a hidden size of 1,024 and 340 M trainable parameters. The size of the syntactic distance embedding is set to be the same as the hidden size of the transformer. We apply a dropout of 0.1 to all the transformer and FNN layers. We select a learning rate of 3e-5 for the transformer layers and 1e-4 for the other layers. We use AdamW (Loshchilov & Hutter, 2018) to optimize the model.

Experimental results

Table 2 illustrates the relation extraction performance improvement from applying entity-centric dependency trees (EDT) to three of the most popular pre-trained language models: BERT (Devlin et al., 2019), ELECTRA (Clark et al., 2019), and RoBERTa (Liu et al., 2019). All three models use the base scale model (110M). By using EDT, all models show performance improvements ranging from 0.4 to 1.7 points, with RoBERTa exhibiting the most noticeable performance improvement. Language models can enhance the contextual understanding of input sentences by pre-training on massive amounts of text data. Nevertheless, the performance observed with EDT indicates that EDT provides information beyond what language models can acquire alone. This demonstrates that the syntactic and lexical information from EDT has a significant effect on extracting relationships between entities. Since the performance improvement is most pronounced with RoBERTa, comparisons with other models are conducted using the combination of RoBERTa and EDT.

We compare the results of the proposed method to those of previous models using the TACRED, Re-TACRED, and SemEval 2010 Task 8 datasets. The models used for comparison are the nine dependency-based methods and 15 LM-based methods listed below.

Dependency-based methods: C-GCN (Contextualized Graph Convolutional Network) (Zhang, Qi & Manning, 2018), AGGCN (Attention Guided Graph Convolutional Network) (Guo, Zhang & Lu, 2019), BT-LSTM (Bidirectional Tree-structured Long Short-term Memory) (Geng et al., 2020), DPRI (Data Partition and Representation Integration) (Zhao et al., 2021), GMAN (Gated Multi-window Attention Network) (Xu et al., 2022), DDT-REM (Relation Extraction Model with Dual Dependency Trees) (Li et al., 2022), MDR-GCN (Graph Convolutional Network with Multiple Dependency Representations) (Hu et al., 2021), ADPGCN (Attention Dual-Path Graph Convolutional Network) (Wang et al., 2023), DAGCN (Dual Attention Graph Convolutional Network) (Zhang et al., 2024)

LM-based methods: TRE (Transformer for Relation Extraction) (Alt, Hübner & Hennig, 2019), Indicator-aware (Tao et al., 2019), EPGNN (Entity Pair Graph based Neural Network) (Zhao et al., 2019), Entity-aware (Wang et al., 2019), ERNIE (Enhanced language RepresentatioN with Informative Entities) (Zhang et al., 2019), KnowBERT (Knowledge enhanced Bidirectional Encoder Representation from Transformers) (Peters et al., 2019), BERT-MTB (Bidirectional Encoder Representation from Transformers - Matching The Blanks) (Soares et al., 2019), CP (Peng et al., 2020), SpanBERT (Span Bidirectional Encoder Representation from Transformers) (Joshi et al., 2020), DeNERT-KG (Compound of Deep Q-Network, Named Entity Recognition, BERT, and Knowledge Graph) (Yang, Yoo & Jeong, 2020), LUKE (Language Understanding with Knowledge-based Embeddings) (Yamada et al., 2020), Typed-marker (Zhou & Chen, 2022), RECENT (RElation Classification with ENtity Type restriction) (Lyu & Chen, 2021), DeepStruct (Wang et al., 2022a), GPT-RE (Chat Generative Pre-trained Transformers (ChatGPT) based Relation Extraction) (Wan et al., 2023).

Table 3 shows the performance of the proposed method compared to other existing models on TACRED. In dependency-based methods, CGCN and AGGCN are representative models using hard pruning and soft pruning, respectively. When comparing the two models, AGGCN demonstrates slightly better performance, with a 0.8 point improvement. This suggests that preserving lexical information between entities through soft pruning is more effective for relation extraction than resolving long-distance dependencies between entities via hard pruning. The proposed method with EDT significantly outperforms AGGCN and exceeds the performance of all dependency-based methods. This indicates that the proposed method effectively utilizes both syntactic and lexical information for relation extraction by mitigating the trade-off between them. Additionally, the impact of the deep contextual representations provided by the language model is substantial. In fact, when comparing dependency-based methods with LM-based methods, most LM-based methods exhibit superior performance. Among LM-based methods, the proposed method stands out in terms of performance. Particularly, when compared with models using base-scale language models such as TRE, ERNIE, CP, Know-BERT, and DeNERT, the EDT RoBERTa-base model of the same scale achieves the highest performance. In some cases, our model outperforms the other models with extra training data which are ERNIE, LUKE, Know-BERT, DeNERT and BERT-MTB. Even the EDT RoBERTa-base model outperforms LUKE by 0.5 points despite being smaller in scale. The LUKE model additionally fine-tunes the RoBERTa large model (340M) with a large amount of entity-annotated corpus obtained from Wikipedia. Furthermore, in terms of training costs, such as the amount of corpus and re-pretraining time, our model is more efficient since it only requires dependency parsing. However, We find that the proposed method shows lower performance than RECENT and DeepStruct. In the case of RECENT, several specific classifiers are trained based on combinations of entity types. Due to its strong dependency on entity types, it has the drawback of being challenging to use when data lacks entity types or when a new entity type appears in the data (e.g., SemEval 2010 Task 8). In the case of DeepStruct, a model with 10 billion parameters is used, and additional data is also utilized. Therefore, the high performance of DeepStruct is due to the large parameter size and the use of extra data. In summary, while the proposed method requires a dependency tree, it is far more efficient compared to other models that utilize additional data and large-scale models. This indicates that, rather than merely using large datasets and models, it is essential to research effective ways to utilize appropriate features even in LM-based methods.

Table 4 illustrates the performance comparison on Re-TACRED. The proposed method shows the best performance with an F1 score of 91.2%. When compared with dependency-based methods, it is evident that the proposed method achieves the highest performance, similar to the results on TACRED. Additionally, even when compared to LM-based methods using language models of the same scale, the proposed method still demonstrates superior performance. Compared with the experimental results on TACRED, the proposed method shows an F1 score improvement of 15.3 points. This is a larger improvement compared to the SpanBERT-large model’s improvement of 14.5 points, indicating that the higher the integrity of the data, the better the efficiency of EDT.

Table 5 illustrates the performance comparison on the SemEval 2010 Task 8 dataset. Our model shows good performance on the SemEval 2010 Task 8 dataset as well. Among dependency-based methods, GMAN, like the proposed method, utilizes information from the dependency tree in conjunction with the language model. GMAN also demonstrates higher performance than most LM-based methods on the SemEval 2010 Task 8 dataset. Similar to experiments on other datasets, the proposed method shows higher performance than models that use additional datasets, such as KnowBERT and BERT-MTB. This further proves that increasing the size of the language model and the training data is not the only solution. Indicator-aware defines a syntactic indicator that is syntactically important in a sentence, which shows good performance in relation extraction. The syntactic indicator is manually designed based on the results of morphological analysis and named entity recognition. The attention distribution β in Eq. (6) plays a similar role to this syntactic indicator. Unlike Indicator-aware, our proposed method automatically calculates β through training. This is a further advantage of the proposed approach. Compared to TACRED, the performance of GPT-RE has improved significantly on SemEval. GPT-RE is a model based on GPT-3 (Brown et al., 2020). Considering the massive number of parameters in GPT-3 (175 billion parameters), it is clear that this approach lacks consistency and efficiency. The proposed method adopts the size of RoBERTa Large (340 million parameters). In terms of efficiency, it is evident that the proposed approach, even at roughly 1/500th the size, can achieve comparable performance, making it better suited for relation extraction.

Analysis

To measure the effectiveness of each component in the proposed method, we conducted an ablation test, as presented in Table 6. We found that the GAT contributes to an improvement of 0.3 points, as it effectively acquires overall lexical information. The F1 score dropped by 0.8 points when we removed the SDA, resulting in an F1 score of 72.4%. This indicates that the SDA significantly contributes to acquiring syntactic information from the entity-centric dependency trees. When both GAT and SDA are removed, the performance drops by 1.7 points. This decline is greater than the sum of the declines caused by each component individually, suggesting that the two components not only complement each other but also have a synergistic effect. Although the pre-trained language models have abundant contextual knowledge, we observed that their performance can be further improved with additional lexical and syntactic information.

Table 7 presents the experimental results applying the representative hard pruning method, C-GCN, and the soft pruning method, AGGCN, to the RoBERTa-base model for a fair comparison between dependency-based methods and the proposed method. The ‘w/o dependency tree’ indicates the performance of using only the RoBERTa-base model for relation extraction without utilizing a dependency tree. Compared to this baseline, C-GCN and AGGCN show performance improvements of 0.4 points and 0.7 points, respectively. The proposed method achieves the highest performance improvement with an increase of 1.7 points. The performance difference between C-GCN and AGGCN is attributed to the utilization of lexical information. Since C-GCN is based on hard pruning, it can overlook important lexical information crucial for relation extraction. In contrast, AGGCN, based on soft pruning, reduces the likelihood of this issue occurring. The proposed method has the advantage of harmoniously utilizing both lexical and syntactic information compared to the two methods. The performance gap shown in Table 7 is attributed to these characteristics.

To check whether syntactic information and lexical information are effectively learned through the proposed neural network architecture, we visualized the attention weights (i.e., α + β) of each token through heat maps, as shown in Fig. 4. In the case of the first sentence, syntactic information from words between ‘Pauliina Miettien’ and ‘Sky Blue FC’ is sufficient for relation extraction, and the attention weights are well focused on words that constitute the SDP of the two entities. However, the sole use of SDP does not provide enough information for relation extraction in the second sentence. Fortunately, the attention weights are distributed not only among the SDP but also among key phrases outside the SDP. Hence, the model was able to predict the spouse relationship accurately. Since TACRED is a semi-automatically annotated dataset, we observed instances with mislabeled target relation tags. In cases such as the third sentence, the proposed method predicted labels that are more appropriate for the ground-truth labels. The fourth sentence is an example of a typical off-target prediction by the proposed method. The attention weights are mainly distributed on words such as ‘looked’ (the LCA of the two entities), ‘father,’ and ‘daughter’ (information necessary for relation extraction). However, for relation extraction, the model should be able to locally reason that the subject entity ‘his’ refers to ‘Knox’s father’ or ‘Curt Knox,’ and the object entity ‘she’ refers to ‘his daughter.’ Consequently, further study on complex information (e.g., coreference resolution) and reasoning beyond lexical information and syntactic information is needed for complete relation extraction.