All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
Following the last round of revisions, and based on the reassessment of one of the original reviewers, I recommend acceptance of this submission in its current form.
[# PeerJ Staff Note - this decision was reviewed and approved by Xiangjie Kong, a PeerJ Section Editor covering this Section #]
After the revision, the previous confusing part has been improved. The problem, technique, and results are all clearly and concisely communicated in this work. The work mainly focuses on data scarcity issues in the Relation Triplet Extraction (RTE) zero-shot scenarios. The motivation, details of the methods, and the contribution are all illustrated properly.
The authors select several popular datasets like WikiZSL and FewRel to evaluate the performance of the proposed methods and conduct enough experiments to prove the effectiveness and robustness of the methods. I think the result is convincing and I appreciate the way to combine ontology with language model.
A thorough ablation study evaluates the contribution of each of the five components in KBPT. Additionally, a case study reveals that while the synthetic samples exhibit reduced word diversity, they possess increased entity diversity compared to real instances. Overall, this methodology offers remarkable validation and transferability, providing valuable insights into zero-shot relation triplet extraction.
None
All three reviewers are generally very positive about the soundness and quality of this work. They do however recommend a set of revisions to improve the paper in terms of clarity in writing and presentation, as well as further discussion of relevant work that they suggest. I encourage the authors to revise the paper following reviewer advice and to provide a response letter addressing them point by point.
**PeerJ Staff Note:** It is PeerJ policy that additional references suggested during the peer-review process should only be included if the authors agree that they are relevant and useful.
**PeerJ Staff Note:** Please ensure that all review and editorial comments are addressed in a response letter and that any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate.
**Language Note:** The review process has identified that the English language must be improved. PeerJ can provide language editing services - please contact us at [email protected] for pricing (be sure to provide your manuscript number and title). Alternatively, you should make your own arrangements to improve the language quality and provide details in your response letter. – PeerJ Staff
The paper is well-written in clear and unambiguous professional English, and effectively communicates the problem, methodology, and results. Considering article structure, the paper provides context by highlighting the gap in existing studies related to the application of knowledge representation in zero-shot scenarios for Relation Triplet Extraction (RTE). In this paper, the utilization of ontology, the construction of prompt templates, the generation of synthetic data, and the use of the generated data to make zero-shot predictions are well demonstrated through pictures.
The KBPT proposes an innovative framework for zero-shot relation triplet extraction, addressing data scarcity challenges through a knowledge-based approach. The generation of synthetic data exploits the knowledge inherent in LMs. Moreover, the combination of additional ontological information makes the data features be closer to the target domain, which I think is a promising idea to solve the zero-shot\cross-domain extraction problem.
The outcomes of zero-shot setting relation triplet extraction on Wiki-ZSL and FewRel datasets underscore KBPT's commitment to the accessibility and robustness of synthetic data. The examination delves into various facets, including different training data ratios, values of Top-n branch p, and threshold t in triad decoding, shedding light on the nuanced impact of hyperparameters on method performance. The ablation study rigorously assesses the contribution of each of the 5 components in KBPT. The case study illustrates that synthetic samples exhibit a reduced diversity in unique words but a heightened diversity in unique entities when compared to real instances.
In summary, the proposed methodology exhibits commendable validation and transferability, offering potential extensions to the ever-evolving landscape of LMs infrastructure. The authors meticulously explore experimental nuances, providing valuable insights into the realm of zero-shot relation triplet extraction.
Authors looking a the domain of RE by emphasizing the lack of the training data
problem. To cope with this, authors follow the Distant Supervision paradigm
(DM) by overviewing related studies. Authors conclude the major problem:
semantical issue of examples generated automatically for augmenting training
data. Therefore authors propose ontology schema for data generation to cope
semantic issues. The latter counted as KB (Knowledge Base), which is in
further utilized in Prompt-Tuning process. Therefore the name of the system is
KBPT.
Authors follow the idea of the reserved tokens, served for encoding a
task-specific knowledge.
The overall introduction and logic flow makes sense. Some moments from the
prospect of the supportive studies might be significantly enhanced. I left the
personal comments on revision in the "ADDITIONAL COMMENTS" section.
Authors utilize Span-Level Attention Matrx to affect the irrelevant external knowledge.
Therefore authors construct a target-oriented architecture that represents a mixture of the Self-attention with the attention sparcity techinque (ECT) + Masking for structuring content, or attention patterns (https://arxiv.org/pdf/2004.08483.pdf).
I believe that it is worth to cite and mention the related studies, as well as align them with the related math equations.
The result is module SAA (Syntax Aware Attention) imputed into network.
The overeall presentation and material is good. The flow of the
presetnation might be significantly enhanced, partially reduced
once authors provide the analogy with the existed attention
sparcity techniques.
Such claims that affect the results obtained
by embedded components are expected to be mentioned at the
experimential stage of the paper or later.
See more comments in ADDITIONAL COMMENTS section.
Authors exploit virtual tokens in tuning which is mimicking the prompt-tuning
technique. (earlier approaches, known before instructive tuning).
The citations and list of the most-relevant works are known to me as a reviewer.
Authors assess the designed KBPT on 4 datasets.
Statistics per each dataset is provided.
Metrics details provided as well, authors utilize F1.
Authors mention the hyperparameters to the full extent.
Dataset splits (train/validation/test) details are mentioned per each dataset.
The overall training process makes sense and relies on early stopping determined by results on validation set.
The drawbacks are:
Figure 5: it is unclear on which data the accuracy results were obtained.
Similar is to Figures 4.
The flow of analysis and results might be simplified. Analysing the paper it was found that authors prefer to follow NYT dataset for the detailed KBPT investigation.
Figure 9 and the related analsys with ROC-AUC curves contrubtes but seems to be redundant
The organization of the code both in the provided resources as well as the code looks good.
It includes all the necessary package dependencies for reproduction.
Each implementation involves unit-tests.
The code provided in attachment to the paper is aligned with the opensourced version on github.
The repository content is well structured, event provides train/test/dev splits information per each dataset utilited in experiments.
Authors required to provide documentation that would introduce the API for those who running into the KBPT framework.
Minor comments:
lines 55-56: "But it is still based on a fixed number of relation types besides
the poor annotation quality" -- it is better to support the claim of the poor
quality with the related studies, benchmarks by providing the realted
citations. The key concept of the distant supervision is the application of
the rule-based approach to perform annotation which is expected to be mostly
aligned. As it was mentioned, noise is acceptable factor, which is expected to
be leveraged by the amount of correctly annotated data. Similar issue in lines
110-111: "First, a large amount of knowledge noise exists in the external
knowledge base."
lines 60-64: Sentence is too long, could be splitted in parts.
lines 70-72: potentials by prior knowedge -> .. the potential of zero-shot
generation LEVERAGING prior knowledge (enhance readability).
lines 76: Persion -> Person.
lines 88-89: After that, Ye et al. Ye et al. (2022) propose t -> Relying on
studies / Based on studies (enhance readability).
lines 115-117: ". However, constructing a complex prompt template that encodes
multiple triples can negatively impact the generation quality of synthetic
samples as the model needs to process multiple relations at once, which is not
trivial." -- it is expected to see studies that support the negative impact and
if so, then to what extent?
137: we conclude this paper and give an outlook -> conclude studies in this
paper. / or not to mention the fact of conclusion.
196-197: this could be also supported by studies:
https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
218-220: sentnce is incomplete to the generated virtual tokens.
220-230: The concept of the Knowledge-Based prompt tuning could be also
compared with the graph-based and structured information retrieval
https://aclanthology.org/2022.semeval-1.188.pdf
338: Quotes issues.
378-381: This concept is could be cited as sparced widow-based attention mechanism (ETC paper).
385: FFN could be refered to other paper rather studies in Vaswani which are more oriented on encoder-decoder archictecture, equipped with Self-Attention mechanism.
424: Due to the powerful language representation ability of PLM" -- this
sentence is similar to the one in line 196. The weak control of differences
between "language representation ability" and "language understanding
capability".
528-529: Authors mention F1 metric, however the actual introduction to metrics happens later on at 586. F1 at 528 may be counted as unknown metric in non-declared evluation approach.
534-536: We can't claim the performance improvement (line 536) before the actual experiments. Is this statement is related to findings from this or other studies?
734: That is why -> Therefore (writing style correction)
1. The English language should be improved to ensure that an international audience can clearly understand your text. Some examples where the language could be improved include lines 57- 86, 258-261 and some texts in Tables. I recommend focusing on the missing blanks and grammar.
2. The paper introduces the background work but misses some related work that also focuses on schema and ontology, like: Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction.
I thank you for the comprehensive experiments, and the performance is convincing. But the description of the methods is not sufficient. The template and the use of generated synthetic data are confusing in Figure 1 and Figure 3.
No comment
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.