All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
The current revision addresses all of the reviewer comments from the last round, as also indicated by two of the original reviewers. The paper can now be accepted for publication in its current form.
The authors took good care of my comments in the first round of review about literature review and restructuring the paper and I have no further comments and questions.
I don't have any other comments for this section.
I don't have any other comments for this section.
no comment
no comment
no comment
The authors have revised the manuscript as suggested. I have no further comments and recommend accepting this paper.
This paper tackles a timely and important research topic, rumour detection in social media. While both reviewers value this work, they also highlight important limitations that will require substantial revisions before it can be considered. Most importantly, reviewers highlight the need to (1) make the novelty of the proposed model clearer, (2) use more recent, competitive methods as baseline models, as the ones used are old, (3) improve the writing and context provided in the paper, as some inconsistencies and need for justification/references has been identified by reviewers.
[# PeerJ Staff Note: It is PeerJ policy that additional references suggested during the peer-review process should only be included if the authors are in agreement that they are relevant and useful #]
[# PeerJ Staff Note: Please ensure that all review comments are addressed in a response letter and any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate. It is a common mistake to address reviewer questions in the response letter but not in the revised manuscript. If a reviewer raised a question then your readers will probably have the same question so you should ensure that the manuscript can stand alone without the response letter. Directions on how to prepare a response letter can be found at: https://peerj.com/benefits/academic-rebuttal-letters/ #]
[# PeerJ Staff Note: The Academic Editor has identified that the English language must be improved. PeerJ can provide language editing services - please contact us at copyediting@peerj.com for pricing (be sure to provide your manuscript number and title) #]
This paper tends to tackle a very important niche in the topic of rumour detection. Although from the methodological perspective, the dataset and the techniques are novel, there are issues regarding the context that need to be clearly addressed:
1- Several places throughout the paper require citation. It is necessary to refer to the manuscripts that support the claimed statements (e.g., line 65,66,71, etc.)
2- Some of the statements are very strong. For instance, despite what is strongly claimed in line 84-85 as early works, there are plenty of high-quality studies focusing on that approach [3][6]. Such a claim could be agreed upon only if it was supported by a comprehensive literature review or set of references.
3- There is an assumption in this paper that rumour is either unverified or false (The first line of the introduction). Rumours are unverified in some context, and they are not accompanied by substantial evidence for at least some group of people [1] thus if we know that a message is false, then it should not be called rumour anymore.
4- The second paragraph of the introduction is not accurate. What is explained as SDQC support classification is in fact the stance detection which along with rumour detection, rumour tracking and veracity detection constitute the rumour resolution system [2].
5- The second and third paragraphs are semantically disconnected and difficult to follow. The earlier paragraph is about the conceptual phases of the rumour resolution system and the latter one is about the methodological approach (with an emphasis on feature extraction) toward rumour detection.
6- One of the major missing pieces is the gap analysis and the problem formulation. What is expected here is a thorough review of the rumour detection literature in a way that directs readers toward the gap. Here the literature review is unrelated to the gap. For instance, there are several studies on early rumour detection [3] which I expected to see here because this paper aims to flag upcoming rumours as soon as possible without spending time to collect data on the same rumour and that is exactly what early rumour detection system does. Another topic that is expected to be investigated in the literature review is cross-domain rumour detection [4][5]. One of the other missing approaches is rumour detection using context-independent features. Such features are independent of a particular incident and could be used across different domains [5].
7- Another shortcoming is the lack of transparency about the data collection process. Questions such as when did you collect the data? how do the readers retrieve the data and get access to the data points? what keywords did you use to build the queries and collect data? are unanswered. Besides, the events (rumours and non-rumours) are expected to be fully described. The current explanation of the events is quite broad and uninformative.
8- When the dataset is introduced (line 293) the term “instance” is used. Does this term refer to a single message (similar to a tweet on Twitter)? Because not all the readers are Weibo users, it would be helpful to show an example of a post on Weibo visually.
9- Based on my understanding the equivalent terms for Twitter’s reply and retweet in Weibo are comment and repost. If that is correct, then why did you decide to regard the retweets in the PHEME dataset as comments and not repost (line 310-311)?
10- How does the annotation by the Sina Weibo community management centre work (line 159)? What kind of labels a post/datapoint may receive?
11- There are some typos and grammatical issues (e.g,. the first column of Table 1)
[1] Nicholas Difonzo and Prashant Bordia. Rumors influence: Toward a dynamic social impact theory of rumor. Frontiers of social psychology. New York, NY, US:
Psychology Press, 2007, pp. 271–295
[2] A. Zubiaga et al. “Detection and Resolution of Rumours in Social Media”. In: ACM Computing Surveys 51.2 (Feb. 2018), pp. 1–36. DOI: 10.1145/3161603
[3] Kwon, S., Cha, M., & Jung, K. (2017). Rumor detection over varying time windows. PloS one, 12(1), e0168344.
[4] Sicilia, R., Merone, M., Valenti, R., Cordelli, E., D’Antoni, F., De Ruvo, V., ... & Soda, P. (2018, December). Cross-topic rumour detection in the health domain. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (pp. 2056-2063). IEEE.
[5] Fard, A. E., Mohammadi, M., & van de Walle, B. (2020, June). Detecting Rumours in Disasters: An Imbalanced Learning Approach. In International Conference on Computational Science (pp. 639-652). Springer, Cham.
[6] Sicilia, R., Giudice, S. L., Pei, Y., Pechenizkiy, M., & Soda, P. (2018). Twitter rumour detection in the health domain. Expert Systems with Applications, 110, 33-40.
The proposed method is novel and the experiments are well explained; however, there are two issues regarding the robustness of this study:
1- Based on the experimentation setting (line 311-313), you used 3-fold cross-validation. How come you chose three splits here? Why not 5 or 10? You need to justify your decision.
2- For the PHEME dataset, You decided to use #Ferguson unrest, #Ottawa shooting, #Sydney siege as the training and validation set and #Charlie Hebdo shooting, #Germanwings plane crash for the testing (line 303-307). Like the previous point, this decision is not justified as well. One quick fix for both is to do sensitivity analysis by running new experiments. For the k-fold cross-validation issue, this means to run the new experiments when k=3,5, 10 and show to what degree the results change by increasing the number of splits. For the second issue, it means to use different datasets for training-validation and test and show how much the results are dependent on the choice of train-validation-test sets.
Additionally and as I explained before, a coherent chain of related work, research gap, and research questions are absent in this paper. Hence although the experiments and few-shot learning based approach toward rumour detection is very well explained, they are not based on a crystal clear motivation (which comes from an in-depth literature review and subsequent gap identification)
The methodological aspect of this study is quite novel and tends to address a very important challenge in automatic rumour detection systems.
See General Comments for the Author.
See General Comments for the Author.
See General Comments for the Author.
The spread of rumors will cause the panic of the public and place considerable losses on the economy and other aspects of society. To solve the rumor detection problem on social media, the authors proposed a few-shot learning-based multi-modality fusion model named COMFUSE, including text embeddings modules with pre-trained BERT model, feature extraction module with multilayer Bi-GRUs, multi-modality feature fusion module with a fusion layer, and meta-learning based few-shot learning paradigm for rumor detection. Although the writing is unambiguous, this paper lacks sufficient experiments to verify its contribution. Some concerns are listed as follows:
1. The authors should illustrate the innovation of the proposed model. The modules used in this paper are all based on existing models such as BERT, Bi-GRUs, without any innovative technologies proposed in this paper.
2. The latest baseline for comparison in this paper was proposed in 2018, the authors need to compare the proposed method with more recent baselines.
3. In many related works, such as Bian et al. 2020, Ma et al. 2019, and Liu et al. 2019 cited in the paper, have a rumor early detection experiment. They use very few tweets posted before the early detection deadline as the training set, the models proposed in these papers are tested on the test set, and good detection effects are also obtained. I think the method proposed in this paper should compare with these methods.
4. This paper uses a pre-training model to improve the accuracy of rumor detection. I wonder if this BERT model can be applied to other methods based on textual content of tweets, and can these methods also be significantly improved, even more than the Bi-GRUs based model proposed in this paper?
5. The author should use experimental results to show that rumor detection results are insensitive to different pad sizes of posts and comments.
6. Do not use the same notation for different definitions in the paper, such as b and T.
7. In Table 1, why are MH370, College entrance exams, …, Rabies COVID-19 relevant, and Zhong Nanshan, Wuhan are irrelevant?
In short, the writing is clear but the model lacks innovation. And this paper lacks sufficient experiments to verify its contribution. I suggest that the paper should be greatly modified to make it more acceptable.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.