Categorization of tweets for damages: infrastructure and human damage assessment using fine-tuned BERT model

Data pre-processing

The following pre-processing steps are employed before providing data to the fine-tuning process and extraction of TF-IDF, and word2vec features.

1. Removal of hashtags, HTML tags, mentions, punctuations, URLs, and numbers.

2. Conversion of tweets to lowercase.

3. Replacement of the emoji/emoticons with their corresponding text.

4. Fixing the issue of misspelled words.

5. Decoding of abbreviations (thnx, thx, btw, pls, plz etc.).

6. Removal of stop words (Only for TF-IDF and word2vec).

Fine-tunning BERT

The BERT model was introduced by Devlin et al. (2018) at Google Lab and it has proven its significance for a variety of text-mining tasks in several application domains (Malik, Imran & Mamdouh, 2023). The benefits of BERT include faster development, automated feature generation, reduced data requirements, and improved performance. It has two architectures and we are interested in fine-tuning the pre-trained BERT model for damage assessment tweet identification task for binary as well as multi-class classification. The BERT model is pre-trained on a large corpus of English data in a self-supervised fashion and uses the context of language in both directions. Furthermore, BERT was pre-trained on next-sentence prediction and masked language modeling objectives.

To fine-tune the BERT base uncased model (https://huggingface.co/bert-base-uncased), some important steps are required. After applying the above-mentioned pre-processing steps, data transformation and training classifier steps are executed.

Data transformation: To transform data into a predefined format, we tokenized each tweet text into N tokens by the uncased BERT tokenizer. The list of N tokens is modified by adding the [CLS] token at the beginning and the [SEP] token at the end. Then an input representation based on pre-trained BERT vocabulary is generated for each token. The [CLS] token generates word embeddings that derive the input data for classification.

We have chosen 64 and 128 sequence lengths because a maximum of 280 characters are allowed in a tweet and 128 sequence length is enough to handle the most lengthy tweets. Therefore, all tweets are padded up to the length of 64 and 128. After that, attention masks are added to locate real and padded tokens. The vector output of attention masks is then fed to the BERT model and fine-tuning step is performed.

BERT classifier training: There are seven disasters in the CrisisMMD dataset. For training and validation of BERT classifiers, we split each disaster into 80-20 ratios using the stratified sampling approach. After that, we took 80% data from each disaster and combined them to make the training dataset. The remaining 20% of data from each disaster is used for testing the BERT classifier on that specific disaster data. Furthermore, the combined 80% data is further divided into a 90-10 split, in which 90% is used for training and 10% is used for validation. We utilize the BERT base model which contains 12 transfer layers, 12 attention heads, and 768 hidden layers. All entities (class labels, token ids, and attention masks) are combined into one set.

For the classification of damage tweets, we attach the outputs of BERT (after fine-tunning) with an additional layer consisting of Softmax classifier as shown in Fig. 3. We denote Ti as the final hidden vector for ith token and h as a final hidden vector of [CLS] token. This h and all Ti vectors are fed to the feed-forward layer containing the Softmax classifier to get the predicted labels. For fine-tuning, the hyperparameters are chosen by using the guidelines of prior studies Devlin et al. (2018). The batch sizes of 8, 16, and 32 are explored with sequence lengths of 64 and 128 but we added the best results (Section ‘Fine-tunning of BERT and comparison with baselines (damage vs non-damage)’) which are obtained by using 32 batch size. For each epoch, we use the optimizer method to update the parameters, and the output of the trained model is evaluated on the validation set by calculating validation loss, validation accuracy, and validation f1-score. Overall, six BERT classifiers are fine-tuned and we use the Google Colab platform for experimental setup. The detail of hyperparameters is presented in Table 2.

Catastrophic forgetting: The literature demonstrates that while fine-tuning a language model to learn new knowledge, previously learned knowledge may be lost because we unfreeze weights. Researchers Sun et al. (2019) call it catastrophic forgetting in transfer learning and every transformer model is prone to this effect. A range of learning rates is explored to get insights and examine the effects of learning rate on catastrophic forgetting while fine-tuning BERT. The learning rates are 1e−4, 1e−5, 2e−5, 3e−4, 3e−5, 5e−5 respectively. In the training process, all layers of BERT are unlocked so that weights can be updated in all layers during the fine-tuning cycle. After repeating the training process several times and careful monitoring, it allows us to select our starting learning rate. We concluded that fine-tuning with higher learning rates (3e−4, 3e−5, and 5e−5) could lead to convergence failure. The best performance was observed with a learning rate of 2e−5 and this lessens the risk of catastrophic forgetting in fine-tuning.

Overfitting: How to choose the appropriate number of epochs for fine-tuning? It is a common issue for fine-tuning the transformers and deep learning models. So many epochs result in overfitting problems whereas very few may cause under-fitting. There are several methods for selecting an appropriate number of epochs, one can start with a large number of epochs and can stop the training process when no improvement is observed on the selected metric. In this research, we use validation loss as a measure to monitor the performance of the BERT classifier. We concluded that four epochs are an appropriate number to avoid overfitting issues.

Word2vec

Word2vec is an algorithm that is used to generate “distributed word representations” inside a dataset (Ali, 2019). In addition, it can generate a vector of a specific length for each word by taking a sentence as input. Word2vec has demonstrated significant performance in similar NLP tasks (Ali & Malik, 2023; Hussain, Malik & Masood, 2022; Younas, Malik & Ignatov, 2023). The skip-gram and continuous bag of words (CBOW) are the two algorithms supported by the word2vec model to generate word embeddings. We are interested in using the skip-gram model to generate embedding features. The skip-gram model tries to predict relevant contextual words for an input word. Window size is another parameter used to confine the number of context words in a frame and we use a window size of 100 dimensions.

TF-IDF

TF-IDF is a statistical approach to evaluate the significance of a particular word in a large context of the document. This technique is commonly used in NLP and information retrieval (IR) tasks (Malik, Imran & Mamdouh, 2023). It is a weighting technique and the weight of a word in a document is proportional to its frequency of occurrence whereas it is also inversely proportional to its frequency in all documents.

Dataset

This study used a benchmark publicly available dataset, i.e., CrisisMMD (Alam, Ofli & Imran, 2018), to test the effectiveness of the proposed framework. The dataset consists of information about seven natural disasters like floods, earthquakes, wildfires, hurricanes, etc. There are seven disaster files in the CrisisMMD dataset. Each disaster contains tweets related to a specific type of event/disaster that occurred at particular/different locations. The tweet text describes human and infrastructure damages. Originally, the tweets of the dataset had several types of class labels like displaced people, affected individuals, etc. As described earlier, we address damage assessment tweet identification at two levels: binary (damage vs non-damage) and multi-class (infrastructure damage vs human damage vs non-damage) classification. The final labels of tweets are derived as follows:

• Infrastructure damage class: In this class “infrastructure damage, utility damage, vehicle damage & restoration, and casualties” are combined.

• Human damage class: In this class “affected individuals, injured or dead people, missing, trapped or found people, displaced people, and evacuations” are combined.

• Damage class: In this class “infrastructure and human damage classes” are combined.

• Non-damage class: Tweets that describe no damage.

After the compilation of the above-mentioned categorization on seven disaster files, the final form of the dataset is described in Table 3.

Experimental setup

We used Python language to calculate the results. Four evaluation metrics are chosen to evaluate the performance of the fine-tuned BERT, comparable models, and five baselines. The metrics are precision, recall, accuracy, and f1-score. In addition, the Wilcoxon signed rank statistical test (Woolson, 2007) is used to determine whether the improvements are statistically significant or not. Six state-of-the-art classifiers are chosen and comparable models are designed to compare the performance of the fined-tuned BERT model. The classifiers are random forest (RF), logistic regression (LR), support vector machine (SVM), CNN, LSTM, and Bi-LSTM. The reason why we chose these ML and DL models is that they presented a significant performance in similar NLP and text mining tasks (Malik et al., 2023; Rehan, Malik & Jamjoom, 2023). The following comparable models are designed:

1. Word2vec+RF

2. Word2vec+LR

3. Word2vec+SVM

4. TF-IDF+RF

5. TF-IDF+LR

6. TF-IDF+SVM

7. TF-IDF+LSTM

8. TF-IDF+Bi-LSTM

9. TF-IDF+CNN

Furthermore, to compare the performance of the proposed framework with benchmark studies, we have chosen the following studies from the literature.

1. Rudra et al. (2018) used syntactic and low-level lexical features with an SVM model for binary and multi-class classification of damage tweets and evaluated their methodology on the CrisisMMD dataset. This is one of the baselines for comparing fine-tuned BERT performance in binary and multi-class classification tasks.

2. The authors in Madichetty & Sridevi (2021) used syntactic, low-level lexical, and top-frequency features with a weighted SVM classifier. Binary and multi-class classification frameworks are designed for the identification of damaged tweets.

3. Alam, Ofli & Imran (2020) proposed a system for damage assessment tweets identification and we chose it to compare for binary classification. They used bag-of-words features with the RF model.

4. Kumar, Singh & Saumya (2019) explored the impact of TF-IDF with several ML and DL models for the identification of damage assessment tweets as a binary classification. The best results are chosen for the comparison.

5. Lastly, Alam et al. (2021) used CNN and FastText embeddings to design a binary classification system for damage tweet identification. We compared this study with the proposed framework for binary classification.

Fine-tunning of BERT and comparison with baselines (damage vs non-damage)

In this section, we performed fine-tuning of BERT for the binary classification task (damage vs non-damage) and then compared its performance with baselines. The results of experiments by applying sequence lengths of 64 and 128 and batch size of 32 are presented in Table 4. Although we tried eight and 16 batch sizes for these experiments, we obtained the best results with a 32 batch size so we only reported results with 32 batch size. The experiments are conducted using a learning rate of 2e−5, Epsilon of 1e−8, and four epochs. For each sequence length and epoch; training loss, validation loss, validation accuracy, validation f1-score, and training & validation times are reported.

In the first step, the BERT base model was trained and validated to classify the tweets into damage or non-damage using 64 sequence length and 32 batch size for four epochs. The training and validation loss is presented in Fig. 4 (left side). The training and validation outcomes in the form of training and validation loss, validation accuracy, and validation f1-score are presented in the upper part of Table 4. The training loss increases up to epoch 2 but then starts decreasing continuously up to epoch 4. In contrast, validation loss continuously increases from epochs 1 to epoch 4, indicating that further training could overfit the BERT model. The best values for validation accuracy and f1-score are obtained on epoch 2, i.e., 91.20% and 89.28%.

In the second step, BERT was again trained and validated for four epochs using 128 sequence length and 32 batch size and results are reported in the lower part of Table 4. The training and validation loss is presented in Fig. 4 (right side) . The validation accuracy of BERT classifiers is higher than with 64 sequence length but validation f1-score is decreased. On epoch 2, the best accuracy is achieved, i.e., 93.33%, and the best f1-score is 86.33%. The training loss decreases steadily and converges to 0.08 value. In contrast, validation loss increases continuously from epoch 1 to 4, indicating that further training is not useful. Hence, validation loss has demonstrated a symmetrical pattern (increasing) for both sequence lengths, and training loss is continuously decreasing.

In the third step, we tested the BERT classifiers (previously trained and validated) on the test parts of the seven disasters. For each configuration (64 and 128 sequence lengths), the BERT classifiers are tested for each epoch, but we only reported the best results against each sequence length for each disaster. The results are shown in Table 5 and each entry includes a confusion matrix and four metric values (accuracy, precision, recall, and f1-score). For the Iraq-Iran earthquake disaster, the sequence length of 128 presented the best performance (95.96% accuracy and 95.12% f1-score). Likewise, for Sri Lanka floods, 93.94% accuracy and 85.13% f1-score are the best values with a sequence length of 128. The best results are obtained on the Iraq-Iran earthquake disaster and the lowest results are achieved on the Hurricane Harvey disaster. Moreover, the sequence length of 128 presented the best results for the first two disasters, and for the remaining five disasters, the sequence length of 64 produced the best results.

In the fourth step, we compared the effectiveness of the fine-tuned BERT with state-of-the-art benchmarks (Alam, Ofli & Imran, 2020; Alam et al., 2021; Kumar, Singh & Saumya, 2019; Madichetty & Sridevi, 2021; Rudra et al., 2018) for damage vs non-damage tweets classification. Five benchmarks are compared with a fine-tuned BERT model for each disaster and results are added in Table 6. The fine-tuned BERT outperformed the five benchmarks in all seven disasters. For the Iraq-Iran earthquake, the highest f1-score achieved by the benchmark is 78.48% and the proposed framework demonstrated a 95.12% f1-score. Thus 16.64% improvement is observed in the f1-score. Moreover, fine-tuned BERT demonstrated significant improvement in accuracy, precision, recall, and f1-score in comparison to five benchmarks for seven disasters. Specifically following percentage of improvements are observed in the f1-score; for Hurricane Irma, 5.37%; for Hurricane Maria, 0.69%; for Hurricane Harvey, 2.61%; for California wildfires, 6.79%; and for Mexico earthquake, 9.54%; This proved the effectiveness of fine-tuned BERT for binary classification of damage assessment tweets and demonstrated better performance than five benchmarks on seven disasters. Among the baselines, the study Madichetty & Sridevi (2021) presented better performance than the other four baselines.

In the fifth step, a statistical test is conducted to determine whether the improvements are statistically significant or not. For this, the performance of the fine-tuned BERT model is compared with the best-performing baseline using Wilcoxon’s signed-rank test (Woolson, 2007) to check the statistical significance of improvements. This test is non-parametric and the null hypothesis can be rejected at the α level. The null hypothesis is that the both models have the same performance. The results of the Wilcoxon signed-rank test are added in Table 7. We compared both models using the macro f1-score for each disaster. The fine-tuned BERT outperforms the baseline for six disasters. The null hypothesis can be rejected on the α = 0.05 confidence level. At first, the difference in f1-scores for both models is calculated and the rank is assigned based on absolute difference values. Then, the sum of ranks is calculated following the criteria of adding all positive ranks at one point and adding all negative ranks at another point. We got R⁺ = 7 + 6 + 5 + 3 + 2 + 4 = 27, and R⁻ = 1, where V_α = 6. As the minimum sum (i.e., 1) is less than 6, we reject the null hypothesis that both models perform equally. Thus, improvement of fine-tuned BERT is statistically significant for binary classification.

Fine-tunning of BERT and comparison with baselines (infrastructure vs human vs non-damage)

In this section, the BERT model is fine-tuned to perform multi-class classification of tweets into infrastructure damage, or human damage, or non-damage categories. After that, the performance of the fine-tuned BERT model is compared with two benchmarks (Madichetty & Sridevi, 2021; Rudra et al., 2018), and five comparable models.

At first, fine-tuning of BERT is performed using the same parameters described in section ‘Fine-tunning of BERT and comparison with baselines (damage vs non-damage)’, but here objective is multi-class classification. The 64 and 128 sequence lengths are used with 32 batch size and a learning rate of 2e−5 is used. The outcomes are measured in the form of training and validation loss, validation accuracy, and validation f1-score. 80% of all seven disaster data is used for training purposes and the remaining 20% is used for testing purposes. Of 80%, 90% is actually used for training and 10% is used for validation. The training and validation results of fine-tuning BERT are presented in Table 8. For a sequence length of 64, the best validation accuracy is 90.83% achieved on epoch 2. Likewise, the best validation f1-score is obtained on epoch 2, i.e., 85.78%. The response of training loss is continuously decreasing indicating that the model is learning steadily during training as shown in Fig. 5 (left side). The validation loss decreases from epochs 1 to 2 and then starts increasing up to epoch 4, representing that further training may lead to overfitting. The lower part of Table 8 shows the results generated using a sequence length of 128. The response of training loss is again decreasing from epoch 1-4, in contrast, validation loss decreases on epoch 2 but then starts increasing up to epoch 4 as shown in Fig. 5 (right side). The highest values obtained for validation accuracy and f1-score are 90.83% and 88.44% on epoch 2. The four BERT classifiers are then tested on the test parts of seven disasters. This completes the training and validation of fine-tuning of BERT classifiers for multi-class classification.

After that, we tested the BERT classifiers on the test part of each disaster and the best results are reported in Tables 9 and 10. In addition, the performance of the fine-tuned BERT model is compared with two benchmarks (Madichetty & Sridevi, 2021; Rudra et al., 2018), and five comparable models. There are nine comparable models but we added results of the top five comparable models. The models are trained on 80% data and then tested on the remaining 20% of the disaster datasets. Table 9 shows the accuracy and per class (infrastructure vs human vs non-damage) f1-score values with macro-average. For the Iraq-Iran earthquake disaster, the fine-tuned BERT outperformed and obtained accuracy (93.88%), precision (85.40%), recall (80.66%), and macro-f1-score (82.63%). The improvement of 6.13% in macro-f1 is obtained by the proposed framework as compared to the benchmark (Madichetty & Sridevi, 2021). Moreover, we can notice the improvement in human damage and non-damage classification presented by the proposed framework as compared to benchmarks and five comparable models. An improvement of 16.20% in human damage and 19.37% in non-damage category classification is observed. Likewise, for macro-precision and macro-recall metrics, it is evident from Table 10 that 4.5% and 5.79% improvement is achieved by the proposed framework compared to the benchmark (Madichetty & Sridevi, 2021).

For the Sri Lanka flood disaster, the proposed framework outperformed the two benchmarks, and five comparable models by demonstrating a 10.33% improvement in macro f1-score, 12.28% in accuracy, 0.53% in macro-precision, and 17.89% in macro-recall measures. Moreover, improvement of 1.25% in infrastructure damage, 17.2% in human damage, and 12.53% in non-damage are observed in the f1-score. For the Mexico earthquake disaster, the proposed framework outperformed all models. An improvement of 6.64% in macro f1-score, 9.2% in macro-precision, and 4.87% in macro-recall are detected compared to benchmarks. Considering the California wildfire disaster, the fine-tuned BERT model delivered the highest performance. We observe an improvement in macro f1-score (12.95%), macro-precision (1.25%), and macro-recall (14.74%). Furthermore, an improvement of 13.16% in human damage, and 26.98% in non-damage classification is obtained by the fine-tuned BERT model.

For the Hurricane Harvey disaster, the proposed framework presented much better performance than state-of-the-art benchmarks and five comparable models. Considering the f1-score, the proposed framework obtained 8.54%, 19.17%, and 6.94% improvements in human damage, non-damage and macro f1-score. Furthermore, 2.33% and 10.89% improvements are observed in macro precision and macro recall. For the Hurricane Maria and Hurricane Irma disasters, the proposed framework performed better than benchmarks and comparable models in accuracy as shown in Table 9. Regarding the f1-score, non-damage classification is improved by 17.7% and 21.07% for Hurricane Maria and Hurricane Irma but did not perform better in the macro f1-score. This shows that fine-tuned BERT is not trained well on these two disaster datasets. This completes the evaluation of the proposed framework on seven disasters for binary classification and multi-class classification.

In the end, the statistical test is performed using the Wilcoxon signed-ranked test to check whether the improvements of the proposed model are statistically significant or not for multi-class classification. As the Wilcoxon sined-ranked test applies to two classifiers, therefore fine-tuned BERT model is compared with the second-best performing model, and the results are reported in Table 11. For justifying the improvements to be statistically significant, our analysis should reject the null hypothesis. The performance is compared in macro f1-score for each disaster and fine-tuned BERT outperformed the second-best for five disasters. After calculating the difference in performance, the ranks are assigned using absolute values. Then the sum of ranks is calculated and we got R⁺ = 25 and R⁻ = 3, where V_α = 5. By comparing R⁻ and V_α, the former is less than the latter and we reject the null hypothesis. Hence, the improvements of the fine-tuned BERT model are statistically significant for multi-class classification.