All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
All concerns raised by the reviewers have been satisfactorily addressed, and I am pleased to inform you that your work has now been accepted for publication in PeerJ Computer Science.
Please be advised that you cannot add or remove authors or references post-acceptance, regardless of the reviewers' request(s).
Thank you for submitting your work to this journal. I look forward to your continued contributions on behalf of the Editors of PeerJ Computer Science.
With kind regards,
[# PeerJ Staff Note - this decision was reviewed and approved by Sedat Akleylek, a PeerJ Section Editor covering this Section #]
N/A
N/A
N/A
**PeerJ Staff Note:** Please ensure that all review and editorial comments are addressed in a response letter and that any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate.
**Language Note:** The review process has identified that the English language must be improved. PeerJ can provide language editing services - please contact us at [email protected] for pricing (be sure to provide your manuscript number and title). Alternatively, you should make your own arrangements to improve the language quality and provide details in your response letter. – PeerJ Staff
Analyzing cyberattacks using Cyber Threat Intelligence (CTI) data presents significant challenges due to the data's diverse formats, intricate structure, and massive scale. To overcome these issues, the authors introduce RAGIntel, a Retrieval-Augmented Generation (RAG)-based large language model framework designed to improve analysis accuracy by retrieving and incorporating structured threat intelligence into the reasoning process.
Overall, the paper is well-written and presents a promising approach. However, I have several suggestions to improve the current version:
1) The GitHub repository containing the RAGIntel implementation appears to be inaccessible. Please ensure that the link is active and publicly available.
2) I recommend adding a dedicated "Threat Model" section that clearly outlines the attacker's objectives and knowledge assumptions. This will help readers and reviewers assess the realism and practicality of the attack scenarios.
3) While the paper focuses on using Retrieval-Augmented Generation (RAG) for cyberattack investigation, recent work such as [A] has shown that RAG systems themselves are susceptible to knowledge poisoning attacks. It would strengthen the paper to acknowledge and discuss these vulnerabilities in the context of your approach.
4) The paper would benefit from a report on the computational cost of the proposed method, such as overall running time.
[A] Traceback of Poisoning Attacks to Retrieval-Augmented Generation. In The Web Conference 2025.
no comment
no comment
The paper generally uses professional English, but there are instances of awkward phrasing and grammatical errors that could be improved for clarity.
The introduction sets the context of cyberattack investigation and the limitations of LLMs, motivating the proposed RAG approach. The related work section provides a good overview of RAG performance enhancements and applications in cybersecurity, with relevant references.
Figure 2 provides good examples of CTI types, but the text references it as if it provides a comprehensive overview of all types, which it does not.
The research proposes a novel RAG-based LLM system for cyberattack investigation, which falls within the scope of computer science and cybersecurity.
The methodology employs sound technical approaches (hybrid retrieval, reranking, compression) but has some limitations, such as having a small evaluation dataset (110 queries total), limited to a single benchmark source and no real-world validation.
The underlying data comes from established benchmarks (CTIbenchmark). The evaluation uses multiple LLMs and comprehensive metrics (RAGAS framework), which strengthens the findings.
The comparison of F1 scores between RAGIntel's factual correctness and standalone LLMs, shown in Table 3 vs. Tables 1 and 2, requires a more nuanced discussion. The numbers suggest that RAGIntel, while potentially reducing hallucinations, might not always achieve higher raw factual correctness scores compared to models pre-trained on similar data.
The paper presents a valuable contribution to the field of cybersecurity and LLMs. The RAG-based approach for threat intelligence is timely and addresses crucial limitations of standalone LLMs.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.