All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
I agree with the reviewer's assessment. The paper is acceptable. Some typos can be corrected.
[# PeerJ Staff Note: Although the Academic and Section Editors are happy to accept your article as being scientifically sound, a final check of the manuscript shows that it would benefit from further English editing. Therefore, if you can identify further edits, please work with our production group to address them while in proof stage #]
The paper is well written and clearly organized.
In the revised version of the paper the references have been updated wrt the previous version, including missing papers too, and better organizing the discussion of the cited literature.
The aspect of tables should be improved, first of all by applying to them all the same layout and the same character size.
According to my knowledge, the paper provides a careful investigation, not always systematic, but quite complete and correct.
The contribution that this paper provides to the existing literature is clear and the conclusions well stated.
Notwithstanding the variety of surveys about hate speech detection and related tasks, the paper provides a quite original perspective on this research area.
The paper has been carefully revised bu the authors and the suggestions provided by the reviewers used for improving its quality.
For what concerns in particular the discussion about multi-lingualism, I also appreciate if some words in the final version will be spent about the fact that the most participated task about hate speech detection (Hateval, often cited in other parts of the paper) is in principle a multilingual task since it is based on Spanish and English data collected and annotated according to the same methodology.
The authors are moreover strongly encouraged to apply a final careful spell check because I've detected some typos (listed below) and probably some other occurs that I've not detected in my reading.
line 21: content are > contents are OR content is
line 77: data and corpora resources > data and corpora
line 173: around 30 when > around 30 points when
line 193: a handful of recent studies suggest > a handful of recent studies suggests OR recent studies suggest
line 803: abuse posts > abusive posts
Please address the reviewers' concerns and provide a point to point response.
[# PeerJ Staff Note: Please ensure that all review comments are addressed in a rebuttal letter and any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate. It is a common mistake to address reviewer questions in the rebuttal letter but not in the revised manuscript. If a reviewer raised a question then your readers will probably have the same question so you should ensure that the manuscript can stand alone without the rebuttal letter. Directions on how to prepare a rebuttal letter can be found at: https://peerj.com/benefits/academic-rebuttal-letters/ #]
The paper introduces a survey about the possibility of generalization of existing hate speech detection models and approaches.
The topic addressed broadly belongs to the sentiment analysis area and especially to the application of sentiment analysis techniques to the detection of hateful content and related phenomena, a field which is considered ad highly interdisciplinary and can be of some interest for the journal.
The paper is clear for what concerns the content and well written for what concerns English.
The organization of the paper is clear and allows the reader to easily find the section of possible interest.
The literature of the area has been carefully analyzed and reported including also very recent papers.
The article introduces a rigorous and precise survey of the hate speech detection task under a novel perspective which according my knowledge has never been systematically surveyed by other papers. Previous surveys and papers where the resources considered in this study are properly cited.
The methodology applied is carefully described and can be reproduced by other researchers.
The organization of the paper is that typical of surveys and help the reader in understanding the content of the paper. Moreover also novel proposals for addressing the weaknesses detected in the analyzed studies are provided by the authors showing their involvement in the area and paving the way for further discussing the topics addressed.
The paper analyzes a very interesting topic and provides enough support for the ideas expressed. The analysis is organized around a set of parameters that allows a discussion about all the crucial aspects.
What in my opinion reduces the interest of this paper is the fact that it doesn't take into account the multilingual perspective and, without an explicit citation of the fact that the study only focus on English language, it seems to assume that hate speech detection is only applied on English data. This assumption seems also supported by the citation of the survey about Arabic, as an exception, and by the fact that the possibility of extending the present study in a multilingual perspective isn't even listed in future work section. But this assumption is not correct: there are tasks about hate speech on languages other than English (e.g. for Italian within Evalita or for Spanish within SemEval and IberEval), and also some of the datasets cited and analyzed in the paper are multilingual (e.g. Basile includes data for English and Spanish both).
In my opinion, the generalization across languages is a crucial topic that is currently addressed in a growing set of papers about sentiment analysis and related tasks, and as a review, but also as a reader, it is a topic that I'd expected to find in a survey about generalization of hate speech detection. I suggest therefore a revision of the paper according this direction, by considering language as a further parameter for the evaluation of generalization, citing in the tables the languages involved in the datasets and so on.
I really appreciate the survey and I think that it introduces a perspective of analysis of a very hot task which can be especially interesting for people working on hate speech detection, but also for other researchers because it introduces a methodology that can be usefully applied in other areas.
It should be useful to put table 2 before table 1, thus providing early the references extensively used in the section and necessary to better understand the content of the section.
My major suggestions about how to improve the paper are expressed in previous points.
For what concerns the references, about the exploitation of non standard language in social media there are very recent papers about this topic and among them https://arxiv.org/abs/2011.02063 which provides a multilingual perspective about this topic.
I've detected some typos in the bibliography, where sometimes the first letter in the name of a language or a proper name is not in uppercase as it must be (e.g. english > English, twitter > Twitter, rnns > RNNs)
The paper presents a survey of research works aimed to investigate the generalisability of hate speech detection models. The paper also provides extensive consideration of obstacles and future directions for the task of hate speech.
The article is well-written, clear, and correctly organized. I think the paper is within the scope of the journal and the topic is currently of main interest to the Natural Language Processing community.
The introduction should more explicitly state the audience to which it is referring to and, in order to be more broadly understood, introduce the concept of “generalisation” before using the term (lines 46-48 should be mentioned later in the introduction).
Several literature reviews have been published recently on the subject of hate speech (Poletto et al., 2020; Vidgen and Derczynski, 2020; Fortuna and Nunes, 2018) but none of them cover the issue of generalisation of hate speech detection models. Making the paper an interesting review of a point of view not already surveyed.
The paper presents an unbiased coverage of the subject in the Section named “Generalisation studies in hate speech detection”. While the review comprehends all the papers approaching the problem of generalisation, the research works proposed in this direction are still limited (8 papers). Tables 1 and 2 could be very useful for the community, but the comment associated with these tables is very short and provides a list-based explanation of each of the 8 papers involved. A survey should be able to provide different views and aggregations of the surveyed papers for highlighting commonalities and differences. This is not possible with such a low number of surveyed papers, which makes me doubtful of the need for a survey on this topic at the current stage.
The authors should go more into the details of the surveyed models.
Please present Table 2 before Table 1. The current order makes it not possible to understand that, for example, Basile and Waseem (lines 141 and 142) are actually benchmark datasets associated with papers and not just names. You should also consider using references instead of just the first author’s surname.
The reference (Fersini et al., 2019) is wrong, as it should be associated with “Overview of the EVALITa 2018 task on automatic misogyny identification (AMI)”
The discussion and future directions of the paper should be referred to obstacles and suggestions to address these obstacles for generalisable hate speech detection. However, I find that almost any of the obstacles or suggestions are focused on the generalisation problem but instead on generic hate speech detection. This is a crucial critic of the paper, as the focus on the generalisation problem is the one that makes it different from the existing literature review. The authors should explicitly report, for each obstacle and discussion point, why it is needed for improving generalisation. This is a crucial issue that invalidates the two sections of the paper that are also the longest.
An example of this issue is in the “Reducing overfitting” paragraph, where the authors are suggesting to generalise by training on more datasets as a solution to overfitting. This is problematic because (1) the authors are not mentioning that they are referring to generalisation (which is the topic of the paper) and (2) it is not clear how they are suggesting it while they are also stating that there are big issues when it comes to generalise.
The authors should also comment on the future issues reported in (Fortuna and Nunes, 2018), for example, open-sourcing and multilingual research.
Minor comments:
- Lines 109-114: the paragraph is unclear. Please re-write and clarify what you mean with “section headings of the body of this paper”
- Line 512: Please do not use contractions.
- Lines 558-565: Please consider to cite also the work titled “Comparative Evaluation of Label-Agnostic Selection Bias in Multilingual Hate Speech Datasets” recently presented at EMNLP 2020
- Line 595: Please consider to cite also the work titled “Contextualizing Hate Speech Classifiers with Post-hoc Explanation” presented at ACL 2020
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.