MFEE: a multi-word lexical feature enhancement framework for Chinese geological hazard event extraction

Our approach

In this article, event extraction is formulated as a multi-class classification problem. Based on Nguyen & Grishman (2015), entity types and corresponding event types or argument roles are assigned using the BIO annotation schema.

Our framework for extracting Chinese events consists of three phases: (i) the character representation module that encodes Chinese characters into the real value vector; (ii) the sequence modeling module that can effectively use both past and future input features thanks to a Bidirectional Long Short-Term Memory (BiLSTM) component; (iii) the extractor module, which extracts triggers and arguments using a conditional random field (CRF) layer. Figure 2 shows the architecture of our multi-word lexical feature enhancement framework.

Character representation

The character representation module transforms the character into a real value vector. The output of this module consists of concatenating the following four features:

Character vector features

We transform characters into vector embeddings of distributed representations based on character-level modeling. BERT (Devlin et al., 2018) dynamically extracts character vector features based on context, making it more powerful for characterization. We use the BERT pre-trained model to obtain encoded representations of Chinese characters, which can better characterize the polysemy of characters. Given an input sequence Input = c₁, c₂, …, c_n, where c_i, i ∈ [1, n] is a character in the input sequence, then

x_i = BERT(c_i)

where x_i denotes the vector representation of character c_i obtained by the BERT model.

Word segmentation features

Frameworks based on word-level features alone cannot take advantage of the lexicon. It is easy to lose important local contextual information. Most of the previous work has been done to obtain sequence label results by word segmentation tools to introduce lexical information. The corresponding labels are presented according to the relative position of the current word in the lexicon. Take the “BMES” tag as an example: ${\displaystyle x_i=\left[BERT\left(c_i\right);e^{seg}\left(seg\left(c_i\right)\right)\right]}$ where $seg\left(c_i\right)$ denotes the sub-label of character c_i obtained by the automatic sub-labeling tool, and e^seg denotes the lookup table of the sub-label “BMES”.

Multi-word lexical features

Word segmentation tools can introduce lexical information related to the character context. However, if there is an error in the subsumption effect of automatic subsumption in the current context, it will misuse lexical information. Ma et al. (2019) introduce lexical features for the model by lexical matching in the named entity recognition task. Inspired by it, we use external lexicon matching to present rich multi-word lexical information for characters. This can avoid the accumulation of errors caused by a single lexical segmentation result.

For each character in the input sequence, we construct four sets, namely,{B},{M},{E},{S}, which represent the set of words in the corresponding position of the matched word. B records all words in the lexicon that can match the subsequence starting with the word. {M}, {E}, {S} denote the set of matching words where the word is in the middle of the word, at the end of the word, and a single character, respectively. Take Fig. 2 as an example; suppose the current subsequence fragment is “受灾民众获救” (Affect people are rescued); for the character “灾” in the lexicon, {B} matches “灾民” {M} matches “受灾民” and “受灾民众” {E} matches “受灾” and {S} matches “灾” If the set is empty, it is represented by a particular open word. The four groups are constructed as follows:

${\displaystyle B\left(L\right)=w_{i,j},\forall w_{i,j}\in D,1\le c=i<j\le n}$ ${\displaystyle M\left(L\right)=w_{i,j},\forall w_{i,j}\in D,1\le i<c<j\le n}$ ${\displaystyle E\left(L\right)=w_{i,j},\forall w_{i,j}\in D,1\le i<j=c\le n}$ ${\displaystyle S\left(L\right)=w_{i,j},\forall w_{i,j}\in D,1\le c=i=j\le n}$ where D denotes all words in the lexicon that can be matched by the subsequence containing the character with subscript c.

Each word in the set is first encoded, and then the lexical in the set is represented in an aggregated manner. If a model dynamic weighting method is used to determine the word weights, it tends to increase the model complexity and the loss of time performance. Considering the temporal performance of the method, the frequency of occurrence in the lexicon is used as its weight. The lexicon can be the training set or the text of a vertical domain that belongs to the task requirements. The aggregated representation of each lexicon is then obtained as follows:

${\displaystyle V^s\left(A\right)=\frac{4}{H}\sum _{w\in A}{h}_w{e}^w\left(w\right)}$ ${\displaystyle H=\sum _{w\in B\cup M\cup E\cup S}{h}_w}$ where $V^s\left(A\right)$ denotes the aggregated representation of word set A. h_w denotes the weight of word w, that is, the frequency of occurrence in lexicon. e^w(w) denotes the vector representation of finding word w from the lexicon, while H denotes the sum of all word frequencies of the four-word sets. After the introduction of the multi-word lexical feature, the new character representation x_i is

${\displaystyle x_i=\left[BERT\left(c_i\right);e^{seg}\left(seg\left(c_i\right)\right);e^l\right]}$ ${\displaystyle e^l=\left[V^s\left(B\right);V^s\left(M\right);V^s\left(E\right);V^s\left(S\right)\right]}$

Entity labels features

Entity tagging information can be introduced using word segmentation tools. The entity information can reflect the role that an entity fills in the sequence context. Considering that in most cases, the real data do not have labeled entity labels, we do not directly refer to the labeled entity labels in the dataset as in previous work. Instead, we introduce entity information based on the entity labels of the word segmentation tool. Take the Baidu LAC word segmentation tool as an example. The entity information can be labeled as “person name”, “place name”, “organization” and “time”. The entity label information is introduced according to the entity where the character is located, and the final character is represented as: ${\displaystyle x_i=\left[BERT\left(c_i\right);e^{seg}\left(seg\left(c_i\right)\right);e^l;e^{entity}\left(entity\left(c_i\right)\right)\right]}$ where $entity\left(c_i\right)$ denotes getting the entity label corresponding to character c_i and e^entity denotes the lookup table of entity labels.

Sequence modeling

The Long Short-term Memory (LSTM) model solves the gradient explosion and gradient disappearance problems of ordinary recurrent neural networks while training long sequences. It has good performance in capturing contextual sequence features. Bi-directional Long Short-term Memory (BiLSTM), as a combination of forwarding LSTM and reverse LSTM, can capture sequence semantic dependencies in both directions. Therefore, we obtain contextual features based on BiLSTM. BiLSTM splices the hidden states $\overrightarrow{h_l}$ and ${\overleftarrow{h}}_l$ calculated by the forward LSTM and reverses LSTM to get the remote state of character c_i:

${\displaystyle \overrightarrow{h_l}=\overrightarrow{LSTM}\left(\overrightarrow{h_{l-1}},x_i\right)}$ ${\displaystyle {\overleftarrow{h}}_l=\overleftarrow{LSTM}\left({\overleftarrow{h}}_{l+1},x_i\right)}$ ${\displaystyle h_i=\left[\overrightarrow{h_l},{\overleftarrow{h}}_l\right]}$

Input the obtained sequence representation H = h₁, h₂, …, h_n into the label sequence of prediction trigger words in the Label Inference Layer.

Trigger and argument extractor

This module aims to predict subtypes of character nodes. Our trigger and argument extractor is proposed as a character-level classification task. We use Conditional random field (CRF) to predict labeling results, which takes into account the dependencies between consecutive sequence labels. The conditional random field adds constraints to the final predicted labels based on the information on the dependencies between the labels. These constraints are automatically learned through the CRF layer during the training process. Thus, based on the combined labeling constraints, the conditional random field selects the labeling results with high scores in the labeling list. The higher score indicates the labeling result is more credible, and the labeling with the highest credibility is selected as the final result. It is ensured that the prediction labeling sequence is reasonable and the probability of prediction error is reduced.

Experiment setup

Data Collection with Event Labeling. Considering the quality and authenticity of news texts, we use the news reports related to geological hazards at http://www.chinanews.com as the original corpus. It is used to perform geological hazard-based event labeling. We focus on seven event types: “earthquake”, “collapse”, “landslide”, “debris flow”, “land collapse”, “land subsidence” and “ground fissure”. They belong to the most common geological hazard events in the Classification and codes for natural disasters.

We consider the temporal dimension to classify the events in the field of geological hazards into five major categories. These are pre-hazard, hazard, Influence, Emergency Response, and Achievement events. This is further subdivided into sixteen specific types of events. The definition of events is shown in Table 1 (including trigger words and arguments).

Before the data label, we further preprocess the screened original corpus as follows:

1. Sentence segmentation. We extract information based on the sentence level, such as labeling tasks and model training, so we need to cut the current news text into sentence form. We use “ ∘”, “?”, “!”, “;”, “......” etc. as separators to split the news text into sentences.

2. Rewash the sentence. After the news text is segmented, many new sentences will be generated. The sentence text needs to be cleaned again, such as removing empty penalties, too long or too short sentences, repeated sentences, etc., to ensure the quality of the labeled corpus.

We use manual labeling to label the processed corpus. The labeling operation is simplified by the Jingdong Wise open text labeling tool. There are two stages to the labeling process. Firstly, the event trigger words are labeled, and then the arguments are labeled. We use a multi-person labeling method to increase efficiency and avoid incorrect labeling. We summarize some key parts of our labeling standard as follows:

1. We expect different event types to be represented by different multivariate groups, and trigger words to be considered as unique identifiers of events.

2. We do not only define events as actions or states that change in the real world. Some statements, such as policy notifications, are likewise considered events.

3. We expect the fine-grained labeling of arguments to be complete, but not redundant. Determiners and modifiers for entities are retained only if they have a significant impact on the understanding of the event.

A total of seven people participated in the labeling process (including five labelers and two experts). For each document, we ask the labelers to label it independently. They can consult the experts if they have any doubts. After each batch of documents is labeled, the experts check them. Documents that do not meet the standards are returned for re-labeling. This process is repeated until the acceptance rate reaches 90%. The authors will check again after the expert check is completed. Documents that do not meet the standard are returned for relabeling until the acceptance rate reaches 95%.

Finally, we obtained 673 documents in total, and this number is larger than 633 of ACE 2005 dataset. There are 459 words on average in each document. We divided these documents into train, development, and test sets with the proportion of 8:1:1 based on the time order. Figure 3 shows the number of events of each type on the CGHaz dataset. Figure 4 shows the ratio for each geological hazard type on original documents.

Evaluation Metric. We use the same evaluation criteria as in the previous work. If the predicted trigger word has the same offset as the labeled trigger word, then the trigger word is correctly identified. In the case of the same offset, if the predicted trigger word is classified as the correct event type, the trigger word is proved to be correctly classified. We use Precision, Recall, and F1 values as evaluation metrics for the event extraction task.

Hyper-parameter Setting. For the input, we set the maximum number of sentences and the maximum sentence length as 32 and 128, respectively. We employ the Adam (Kingma & Ba, 2014) optimizer with the learning rate of 0.0015, train for at most 100 epochs and pick the best epoch by the validation score on the development set. We use large-scale Chinese with training word vectors as word vector lookup tables for word segmentation results. And it is used as a lexicon to obtain multi-word feature representation. The Baidu LAC tool obtains word segmentation results and common entity label information.

Performance comparisons

Baselines. We compare the proposed Chinese event extraction model MFEE based on Multi-Word Lexical Feature Enhancement with the following event extraction methods:

1. DMCNN (Chen et al., 2015): Feature extraction based on CNN and retaining the essential information of current trigger words and arguments through the dynamic multi-pool method.

2. C-BiLSTM (Zeng et al., 2016): Fuse the semantic information of words by combining a convolutional neural network and BiLSTM model.

3. BERT (Devlin et al., 2018): An event extraction model based on Bert’s pre-training model connecting task classifier.

4. BERT+BiLSTM+CRF: Bert model extracts the word vector, obtains the sequence context semantic features through BiLSTM, and CRF receives the final event trigger word and argument prediction results.

5. NPNs (Lin et al., 2018): Learning and fusing word and phrase representations using DMCNN. By triggering word generation block and event classification block in concert to complete event detection.

Main Results. As Tables 2 and 3 show, MFEE achieves significant improvements over all baselines on the CGHaz dataset. Figure 5 and Table 4 show the results of argument extraction for the seven geological hazard event types on the CGHaz dataset. Specifically, (1) MFEE improves 1.2 and 1.0 F1 scores overall best baseline BERT-BiLSTM-CRF in event detection and arguments classification, respectively. These improvements are mainly attributed to MFEE’s approach to fusing multidimensional semantic features. It enhances the word vector representation. (2) DMCNN and C- BiLSTM were experimented with based on characters and words. The results show that the word-based model outperforms the character-based model on the event detection task. This reflects the importance of lexical features. (3) Compared with DMCNN and C-BiLSTM, the performance of the BERT-based character-level model is significantly improved. The reason is that the powerful characterization ability of BERT is fully utilized. It also avoids the problem of mismatches between words and triggers words. (4) Introducing BiLSTM for the BERT model to extract sequence contextual semantic features. And use CRF to constrain the output label can improve the performance of event extraction. (5) Although both NPNs and the model proposed by Ding Ling combine character and word information, NPNs only use lexical information from a single word segmentation result. It is susceptible to suffering from word segmentation errors. Ding-Ling’s proposed model uses discard processing when the number of matchable words exceeds 3, which is prone to losing correct lexical information. Our method considers incorporating all matching words. The importance of lexical features is decided collaboratively by a designed lexical feature weighting decision method and word segmentation tools. In this way, accurate semantic information can be obtained.

As shown in Table 5, to verify the generality of MFEE, we evaluate the MFEE framework on the ACE 2005 dataset that contains 633 Chinese documents. We follow the same setup as Zeng et al. (2016), in which 549/20/64 documents are used for training/development/test sets. From the experimental results, the F1 values of our proposed MFEE for trigger word identification and classification on the ACE 2005 dataset are 76.2% and 73.6%, respectively. There is a performance improvement compared with other models. The generality and effectiveness of the MFEE are verified.

Ablation Tests. To verify the effectiveness of our proposed feature enhancement method on the Chinese event extraction task, we conducted a series of comparative experiments on the ACE 2005 dataset. The descriptions of our proposed method and the commonly used lexical introduction methods in previous studies are shown in Table 6. The experimental results are shown in Table 7. Table 7 shows that: (1) Baseline model can achieve 74.3% and 71.0% performance in F1 values for trigger word identification and classification, respectively. Compared with the word-based model, it avoids the mismatch between the word and the triggers word. (2) After introducing the Bi-Char feature representation for the character vector, the F1 values of the model for both trigger word identification and classification decrease. This indicates that Bi-Char does not bring positive semantic information to the Chinese event detection task. (3) Introducing lexical features using a word segmentation tool enhances Chinese event detection. It improved 0.9% and 1.1% on the trigger word identification and classification tasks. Our proposed method of introducing multiple lexical features improves the F1 values by 2.5% and 2.9% for the two tasks, respectively. It indicates that our approach can bring more prosperous and accurate semantic information to the model. (4) We further try to introduce dimensional entity features. It is found that the rest of the metrics of both tasks are improved to some extent, except for a slight decrease of 0.2% in the accuracy rate of trigger word identification. It indicates that introducing entity features can improve the model’s performance.

We conducted comparative experiments on lexical feature importance decisions. The methods to determine the lexical weights are as follows: (1) Word average weighted (Avg-weighted): the average of all lexical representations in the lexicon is taken as the feature vector representation. (2) Word frequency weighted (Free-weighted): the lexical feature representation is obtained by a designed lexical feature weighting decision method. (3) Co-determination (Co-determination): the method adopted by MFEE. The lexical weights in the lexicon are co-determined by using a designed lexical feature weighting decision method and word segmentation results. We introduce multi-word lexical features for the character representation of the Baseline model (Baseline model setup as described in Table 6) by three different decision methods. The experimental results are shown in Table 8. As can be seen from Table 8, the word averaging weighting method brings limited performance improvement to the ACE 2005 dataset. It indicates that the average weighting method does not enable the model to determine the lexical information that is more consistent with the current contextual semantics. We propose a method to collaboratively designed lexical feature weighting decision method and word segmentation tool. The performance of trigger word detection and trigger word classification tasks is improved. It can be seen that the introduction of multi-word feature information requires the selection of a suitable method.