All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
I have read your response letter addressing reviewers' comments. This version incorporates all the changes as expected.
[# PeerJ Staff Note - this decision was reviewed and approved by Xiangjie Kong, a PeerJ Section Editor covering this Section #]
Please incorporate the changes as suggested by the reviewers.
**PeerJ Staff Note:** Please ensure that all review and editorial comments are addressed in a response letter and that any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate.
**Language Note:** The review process has identified that the English language must be improved. PeerJ can provide language editing services - please contact us at [email protected] for pricing (be sure to provide your manuscript number and title). Alternatively, you should make your own arrangements to improve the language quality and provide details in your response letter. – PeerJ Staff
The manuscript demonstrates an overall competent command of English, but the clarity and grammatical precision are sometimes inconsistent. While the core ideas are understandable, some sentences are quite long, loosely structured, and occasionally ambiguous. Some issues include:
- Redundant phrases (e.g., “generated information on renewable energy compared to human experts in this field” – overly wordy).
- Improper use of articles and prepositions ("ChatGPT has been tasked with solutions and interpreting questions”).
- Subject-verb agreement and punctuation inconsistencies occur sporadically.
I suggest a language revision would be advisable to ensure clarity and grammatical precision.
Moving on to the literature references, the paper cites a broad and relevant selection of sources, including foundational works on LLMs (e.g., Vaswani et al., 2017; Brown et al., 2020), NLP applications, and sector-specific studies on ChatGPT in medicine and renewable energy. The literature review is well-populated, yet:
- The depth of engagement with the cited sources is sometimes superficial. Key comparisons with existing benchmarks (e.g., BERT, LLaMA, domain-specific LLMs) could be deepened.
- There is limited critique of prior work. The review is largely descriptive.
I recommend an expansion of critical engagement with cited work and providing clearer connections between existing literature and the novelty of this study.
The article follows a logical and professional structure, with clear sections: Introduction, Related Work, Methods, Results, Discussion, Conclusions (w. Limitations). Key presentation elements are present:
- Figures and tables are used meaningfully (e.g., similarity results, architecture diagrams).
- Algorithm pseudocode is clearly formatted.
- Dataset and code are openly available on Kaggle – a strong point.
However, I can also underline several issues:
- Some figures lack detailed captions and are not discussed thoroughly in the main text.
- Raw response examples are limited; more qualitative insight would strengthen transparency.
I think the paper might benefit from some (minor) corrections: improve figure/table discussion integration, include more response-level examples in the main text or appendix.
From the point of view of the relevance and congruence between results and hypotheses, the paper is largely self-contained and understandable without external materials. The research question (how closely ChatGPT responses align with human expert responses in renewable energy) is well posed and directly addressed using a robust methodology (textual and semantic similarity metrics).
I might just underline several aspects that could be addressed:
- The discussion of results occasionally overstates ChatGPT’s superiority without qualifying limitations.
- The hypotheses are not explicitly stated, even though they are implied.
My recommendation would be to make hypotheses very explicit early on. Add balanced reflections on limitations and avoid over-generalization.
While the paper does not present formal theorems in the mathematical sense, it does define and explain algorithms and similarity measures with sufficient technical detail:
- Definitions of lexical and semantic similarity metrics (Jaccard, Cosine, Levenshtein, Word2Vec) are clearly given.
- Algorithm pseudocode is well-explained.
- Statistical testing (paired t-test) is presented (but could use better statistical interpretation).
The authors might think about several ideas related to the formal results:
- Where statistical analysis is performed, include confidence intervals, effect sizes, and p-values with more rigor.
- State null/alternative hypotheses clearly in formal terms.
This study presents original primary research and is perfectly compatible with the scope and aim of the PeerJ Computer Science journal. It introduces a manually curated dataset of 63 prompts and corresponding expert responses in the domain of renewable energy (crafted specifically for the study). This hand-crafted dataset and its use in a structured, comparative evaluation between ChatGPT and human expert responses represent a novel empirical contribution.
The originality lies in:
- Applying both lexical and semantic similarity measures.
- Triangulating results using a third-party LLM (Gemini) for external validation.
- Offering public access to the dataset and code via Kaggle.
The originality claim is credible and supported by transparent research artifacts.
The core research question ("How accurate and relevant are ChatGPT’s responses in the field of renewable energy compared to domain experts?"). As I already mentioned above, it is implied but not very explicitly stated. Still, it is logically embedded in the abstract, introduction, and methods.
The relevance is high, given:
- ChatGPT’s increasing use in knowledge domains like energy, health, and education.
- A lack of prior systematic evaluation of LLM responses in the renewable energy field, as the authors correctly note.
While the topic is meaningful and the gap well identified, the explicit formulation of research aims and hypotheses is somehow a bit lacking. Making the research objectives and questions more visible (e.g., in bullet form) early in the introduction would improve reader clarity.
Technically, the investigation shows rigor in several key areas:
- Use of multiple text similarity metrics (Jaccard, Cosine, Levenshtein, Sørensen-Dice).
- Application of semantic similarity with Word2Vec embeddings.
- Additional validation via Gemini, offering a second LLM’s perspective on alignment.
However, I can also discern several aspects that could be improved:
- Statistical tests (t-tests) are mentioned but with limited detail (e.g., full test statistics, p-values, assumptions validation, or confidence intervals are underdeveloped).
- Ethical rigor is mentioned but not deeply discussed, aside from stating the use of verified expert responses and avoiding AI-generated noise. No ethical board approval is needed for this type of work, but a brief ethics note on dataset sourcing and AI use would be standard.
In conclusion, I think it is a solid investigation, but the statistical treatment could be more formal, and the ethics discussion slightly extended.
The methodology applied by the authors is described with commendable transparency. The authors:
- Provide pseudocode for both lexical and semantic analysis procedures (Algorithms 1 and 2).
- Detail pre-processing steps like tokenization, stopword removal, and stemming.
- Specify the use of Google’s Word2Vec embeddings and list the semantic similarity formula.
- Describe the construction and validation criteria for the expert dataset.
- Share links to the publicly available dataset and code.
Nonetheless:
- The Gemini evaluation process could use more detail (e.g., how prompts were phrased for Gemini, any response variations).
- Some mathematical notations in the algorithms are inconsistently formatted or unclear to a non-specialist audience.
In general, I consider the methods section is replicable by experienced researchers, although it would benefit from refinement in notational clarity and elaboration of the LLM-to-LLM comparison setup.
The paper explicitly presents original empirical work but does not attempt to claim groundbreaking novelty or high-impact disruption. This is appropriate and aligns with good scientific practice for replication-focused or benchmarking studies. The authors clearly position their study as:
- A proof-of-concept.
- An early-stage exploration into ChatGPT’s alignment with expert responses in a specialized domain.
- A basis for future, broader longitudinal assessments.
They also provide a clear rationale for replication:
- The lack of prior large-scale quantitative evaluation of ChatGPT in the renewable energy domain.
- The need to test and re-test LLM performance over time due to model updates.
- Public availability of data and code supports transparency and reproducibility.
I consider that this modest and responsible framing strengthens the credibility of the work. The justification for replication is clearly articulated and beneficial to the literature, especially in applied LLM performance benchmarking.
As for the assessment of the data provided (robustness, statistical correctness, and control), I can mention the following final ideas:
- Data Availability is clearly met, as the hand-curated dataset and source code are made publicly available via Kaggle, meeting open science expectations.
- Data Quality is also covered, as the prompts and expert responses are sourced from credible, pre-AI online materials, filtered for consensus, clarity, and relevance (an important methodological strength).
- Sample Size: 63 prompts is relatively small for a generalizable study, but suitable for a focused case study. The limitation is acknowledged inside the paper.
- Statistical Analysis is also in order, covering items such as: lexical similarity (multiple standard metrics applied), semantic similarity (computed using Word2Vec + cosine_), and comparative analysis (paired sample t-tests, KDE visualizations, third-party validation using Gemini).
I should also note several shortcomings related to the same topic:
- Lacks confidence intervals, full test statistics, and effect sizes in tables or narrative.
- Some statistical results (especially Gemini scores) are descriptive but not deeply analyzed.
The data are clearly provided and methodologically controlled, but statistical reporting could be improved for reproducibility and rigor.
Finally, in this section, my conclusions regarding the paper are the following:
- Well aligned with the core research objective: evaluating the relevance and accuracy of ChatGPT responses in the renewable energy domain.
- Grounded in the data (including quantitative similarity metrics, semantic alignment scores, and external LLM validation using Gemini).
- Cautiously framed (the authors stop short of making exaggerated claims about ChatGPT’s replacement of experts and acknowledge that this is a preliminary, small-scale study).
The authors also highlight:
- The potential of ChatGPT to support renewable energy education and communication.
- The role of AI as a complementary tool, not a substitute for human expertise.
The conclusions are appropriately cautious, supported by data, and effectively loop back to the (implied) research question. However, the lack of explicit hypothesis framing earlier in the text slightly reduces the overall sharpness of the closing section.
Based on the comprehensive review across all the above reviewing sections, this manuscript represents a well-scoped and timely contribution to the evaluation of LLMs in domain-specific contexts. The study is original, methodologically sound, and presents replicable research with open data and transparent methods. However, some improvements are needed in several areas: the research question and hypotheses should be explicitly stated; statistical reporting lacks maximum rigor (e.g., confidence intervals, test statistics); and the English require a little bit of editing for clarity and precision. Conclusions are appropriately cautious and well-grounded in results. With minor revisions focused on clarity, formal rigor, and better integration of hypotheses, the manuscript would meet strong publication standards.
Acceptable
I have two comments:
1- In my opinion, the list of questions selected as prompts is too general.
Based on the dataset, they are non-technical renewable energy questions; they all focus on basic public awareness topics.
As a result, I suggest adding this to the manuscript. For example, " The systematic evaluation presented in this study is limited to general and non-technical renewable energy questions intended for public understanding, but does not explicitly assess expert energy level questions or engineering content".
2- Also, on the expert's background, "human experts", the manuscript does not specify their qualifications or background, which is a critical omission when the paper is about a systematic comparison. Although in line 148, the author did mention a reference to Li et al. (2023), who conducted an empirical assessment of ChatGPT responses concerning protective action information.
This author stated clearly: "The responses were collected from ChatGPT, and then a survey was developed using these responses. A total of 38 specialized experts in the field of emergency response have assessed the survey and confirmed that the responses generated by ChatGPT were accurate and relevant to the subject area..."
I strongly advise the authors to add this information; without knowing whether the responses came from academics, industry professionals, or general writers, it is difficult to assess the credibility of the baseline used in the study.
Data valid for General Renewable Energy Questions
I want to thank the authors for the tremendous work they have done in updating the manuscript; however, it is important to review section 2. The experimental design needs further clarification.
I recommend major revisions and a re-evaluation of the revised manuscript. This will ensure that the study adheres to the journal’s standards of scientific validity and methodological transparency. If further clarification on specific comments or concerns is required, I am happy to provide additional input.
1. The literature review depends heavily on a limited number of sources, reducing the breadth of context provided. More diverse citations are needed to strengthen the background and demonstrate a wider understanding of the renewable energy and AI fields.
2. The authors mention that the dataset was manually compiled but do not provide sufficient detail on how it was constructed or the criteria used for selecting the 63 prompts. This lack of transparency makes it difficult to assess the quality and representativeness of the data.
3.The results are presented, but there is a lack of depth in connecting them back to the hypotheses and discussing their broader implications in the renewable energy field.
1. The dataset consists of only 63 prompts, which is too small to provide comprehensive insights into ChatGPT's performance. A larger and more varied dataset is necessary for more robust findings.
2. While the paper outlines the basic steps of the experimental design, it lacks sufficient detail on the selection process for the prompts and the overall dataset creation. Without this, the study's replicability is compromised.
3. The study does not address how ChatGPT's capabilities could be applied in real-world renewable energy settings. Adding an exploration of practical applications, such as decision-making tools or educational platforms, would increase the relevance of the research.
1. The dataset used in the study is quite small and narrow in scope, which makes it difficult to draw broad conclusions about ChatGPT's overall performance. As a result, the findings may not apply to more complex or diverse questions within the renewable energy field.
2. The conclusion suggests that ChatGPT outperforms human experts, but this is based on a small, manually selected dataset. Given the limitations of the study, this claim seems a bit too strong, and the conclusion should be more cautious about generalizing ChatGPT’s capabilities.
3. The study looks at ChatGPT’s performance at a single point in time. Since AI models like ChatGPT are continuously evolving, a longer-term study would give a better picture of how its performance might change, especially in a fast-moving field like renewable energy.
No comment
No comment
No comment
-
The paper exhibits a clear and accurate English vocabulary, with a professional approach throughout the entire content. The references section is quite abundant, proving the literature review and the context research at an acceptable level.
The structure of the paper, including chapters and graphics, is clear and unambiguous.
The results/conclusion of the paper are conclusive, related to the hypotheses approached.
The research question, focusing on the capacity of ChatGPT to provide valuable information in the field of renewable energy, is valid and well defined.
The methods and methodology used by the authors is adequately described, in sufficient detail.
The novelty degree is not very high, as in the bases of the paper is actually focusing on the prompts/answers directed to the GPT 3.5 model (~2022), which is already updated by OpenAI to GPT 4.0, GPT-4 Turbo or the latest iteration GPT 4.o.
The provided data, event if not based on the latest tech available, is sound and robust, looking statistically correct and fair.
The conclusion is clearly stated and directly based on the supporting results presented within the paper.
Thank you for submitting your manuscript to the PeerJ Computer Science Journal. The study addresses the critical topic of Large Language Models (LLMs), such as GPT 3.5, and their potential impact on the accuracy and relevance of domain-specific information. However, while the topic is of great significance, the study suffers from notable methodological and contextual shortcomings.
For instance, the manuscript claims to evaluate ChatGPT's performance in renewable energy, yet much of the related work cited does not focus on this domain.
Figures and Tables must be explained before displaying them.
The dataset reported to contain 63 prompts, but it was neither fully explained nor listed, and the examples provided are very basic. The use of such basic prompts limits the depth of analysis and raises questions about the robustness of the evaluation.
The use of GEMINI as an evaluation tool, particularly in Figure 3, is unclear. The suitability of GEMINI for this role is not justified in the text which may affect the validity of the study conclusion.
Furthermore, the claim that "ChatGPT was able to deliver insights on renewable energy in diverse aspects that were superior to the expert’s human knowledge" lacks sufficient empirical support.
Additionally, my testing of the prompts below using both ChatGPT and GEMINI did not yield the same results claimed in the study. For example:
PROMPT 1: "Compare and contrast the capabilities of large language models like ChatGPT with human experts in the field of renewable energy."
ANSWER: "While ChatGPT excels in efficiency, accessibility, and breadth of knowledge, it lacks the depth, context, and practical experience human experts provide. Human experts are indispensable for decision-making, innovation, and addressing the nuanced challenges of renewable energy. ChatGPT, on the other hand, is a powerful complementary tool for research, education, and routine analytical tasks, but it should not replace expert judgment in critical applications. Together, LLMs and human expertise can create a synergistic approach to advancing renewable energy solutions."
PROMPT 2: "Compare and contrast the capabilities of large language models like Gemini with human experts in the field of renewable energy."
ANSWER: "Large language models like Gemini have the potential to revolutionize the field of renewable energy by processing vast amounts of data, identifying trends, and providing consistent analysis. However, they lack real-world experience, creativity, and the ability to consider ethical implications. Human experts, on the other hand, possess invaluable knowledge, innovation, and ethical judgment. By combining the strengths of both, a synergistic approach can be achieved, leading to more efficient, accurate, and sustainable solutions in the renewable energy sector."
These responses differ significantly from the claims made in the manuscript.
Overall, the study addresses a valuable topic but requires significant revisions. Specifically, the authors should:
- Clearly define the selection criteria for human experts.
- Provide detailed explanations and a complete list of prompts used in the dataset.
- Reevaluate the use of GEMINI as an evaluator and substantiate its role.
- Align the related work and research methodology more closely with the study's focus on renewable energy.
- Improve the abstract clearly stating the problem being addressed, the research objectives, the method used and findings.
- Add the research question in the introduction
- Show the answer of the PROMPTS
- Explain why Table 1 is not in the Results
By addressing these points, the manuscript could provide a more rigorous and impactful contribution to the field of LLM evaluation in renewable energy
The manuscript could better explain the rationale for selecting renewable energy as the domain of study. Although the introduction provides a general overview of large language models (LLMs) and renewable energy, it does not sufficiently establish why this domain was chosen over others, nor does it reference recent studies that evaluate LLMs in specialized fields. The absence of citations to benchmark studies on LLM performance in technical domains is a notable gap. Furthermore, while the manuscript structure is sound, there are no figures, tables, or visual data representations included in the main document, which diminishes the clarity of the results. These elements are critical for illustrating comparisons and supporting claims made in the text. The supplemental materials likely contain the raw data and analysis code, but the manuscript does not adequately reference these resources or explain their integration into the study.
To meet the standards of scientific validity, the manuscript must explicitly define all key terms and methodologies. For instance, the use of "similarity measures" to evaluate responses is not clearly defined, leaving readers uncertain about whether lexical, semantic, or hybrid metrics were employed. A concise explanation of these terms, supported by citations, is necessary for the manuscript to stand as a self-contained scientific document. Additionally, formal results such as definitions and evaluations should be elaborated to strengthen the study's validity.
The study aligns with the journal’s aims, focusing on original primary research. The research question—assessing ChatGPT's performance in providing renewable energy information—addresses a relevant and meaningful gap. However, the manuscript lacks detail on how the prompts were designed and curated. The absence of information on prompt selection raises concerns about bias and fairness, as it is unclear whether these were crafted to advantage one participant over the other. The methodology also fails to specify how the human expert was selected and their credentials verified, which is essential for establishing the validity of the comparison.
Although the supplemental files include a Python script and dataset, the manuscript does not describe how these were used or provide sufficient detail for replication. A clearer explanation of the experimental setup, such as how similarity measures were calculated and how human evaluators assessed the responses, is needed to meet the journal's rigorous standards. Transparency in describing these methods is crucial for enabling replication by other researchers.
The findings, while intriguing, are not robustly supported by the data and analyses presented in the manuscript. The authors conclude that ChatGPT performs comparably to a human expert, but this claim is not adequately substantiated. For example, there is no discussion of specific scenarios where ChatGPT's performance differed significantly from the human expert’s, nor are examples of responses provided to illustrate these differences. The absence of such details limits the reader's ability to critically assess the findings.
The statistical methods used to evaluate the similarity between ChatGPT and the human expert are not described in detail. It is unclear whether the analysis included statistical significance testing or confidence intervals to validate the findings. Additionally, the manuscript does not sufficiently address potential biases in ChatGPT's training data or limitations in its ability to generate nuanced, context-specific responses. These omissions weaken the credibility of the conclusions.
The supplemental files include raw data and analysis code, which are essential for reproducibility. However, the manuscript does not provide enough information on these resources to ensure that the findings can be independently verified. A detailed appendix or inline discussion of the dataset and code would enhance the study’s transparency and robustness.
The manuscript would benefit from a more comprehensive discussion of the ethical implications of using ChatGPT in domains like renewable energy. Over-reliance on AI for technical advice could have significant consequences, particularly if the information provided is incomplete or inaccurate. Including such reflections would provide a more balanced perspective and enhance the manuscript's scholarly value.
Additionally, while the study’s focus is on ChatGPT, it could compare its performance to other LLMs or AI systems to provide a broader context for its findings. This would make the manuscript more relevant to the field of AI and natural language processing.
Clearly state the knowledge gap and novelty of the study.
Justify the choice of text-similarity and semantic metrics.
Include statistical tests to determine the significance of findings.
Explain why Gemini was chosen over other alternatives.
While similarity scores are reported, there is no statistical analysis to determine if differences between ChatGPT and human responses are significant. The inclusion of p-values or confidence intervals would strengthen the results.
Figure 4 (semantic similarity scores) could benefit from a more detailed caption explaining the Kernel Density Estimate and its relevance to the study. Additionally, Table 1's caption should clarify the specific purpose of the example prompts shown.
The manuscript introduces several similarity metrics but does not justify why these specific methods (Jaccard, Cosine, etc.) were chosen. Explain why these metrics are most appropriate for evaluating ChatGPT in this context.
The semantic analysis using Word2Vec embeddings is interesting, but the authors should discuss limitations, such as potential biases in the pretrained embeddings.
The limitations section is underdeveloped. The authors should acknowledge potential biases in the evaluation methodology, including the reliance on specific similarity metrics and the absence of real-world applications or stakeholder feedback.
While the language is mostly professional, certain sentences (e.g., lines 73-74 and 88-89) are overly complex and could benefit from simplification.
Narrow Evaluation Framework: The paper uses basic lexical and semantic similarity metrics (Jaccard, Cosine, Levenshtein, Sørensen-Dice) to evaluate ChatGPT's responses. These metrics primarily measure surface-level similarity and do not adequately capture nuanced differences in the quality, depth, or context of information. In state-of-the-art evaluations, more robust frameworks like domain-specific BLEU scores, BERTScore, or human expert-based rubrics are preferred.
Overemphasis on Semantic Similarity:The reliance on Google Word2Vec embeddings for semantic evaluation is outdated compared to advanced contextual embedding models like BERT or GPT-derived embeddings. Modern methods evaluate sentence-level and contextual nuances far better.
Limited Generalizability: The study focuses on 63 prompts, which is insufficient to generalize findings. Current methodologies in evaluating LLMs rely on broader datasets (e.g., thousands of prompts) to ensure statistical robustness.
Comparative Benchmarking Gap: The inclusion of only one other chatbot (Gemini/Bard) for comparison is insufficient. State-of-the-art evaluations typically compare against multiple systems, such as GPT-4, Claude, or domain-specific AI systems.
Ethics and Reliability: While the paper mentions ethical concerns, it fails to critically assess biases in ChatGPT's training data, especially given the reliance on human-generated information that may reflect inherent biases.
Lack of Ground Truth Validation: The "ground truth" data used (human expert answers) lacks a rigorous validation process. High-quality state-of-the-art studies typically involve expert panels, multiple rounds of validation, and domain-specific benchmarks.
Figures and Tables
o Figure 1 (ChatGPT Training Steps): The diagram is overly simplistic and does not add significant value. It lacks technical depth on how transformer models work, such as attention mechanisms or fine-tuning details.
o Figure 4 (Semantic Similarity Distribution): The use of Kernel Density Estimation (KDE) is poorly justified. While it shows a distribution, the lack of comparison against benchmarks makes it unclear whether the semantic similarity values are truly impressive or merely average.
o Figure 6 (Answer Representations): The bar chart comparing ChatGPT and human expert responses is uninformative without error bars or confidence intervals. This makes the reliability of the 99% claim questionable.
o Table 2 (Similarity Metrics): The similarity scores pre- and post-processing barely change (e.g., Jaccard: 0.39 to 0.39, Cosine: 0.60 to 0.61). This raises concerns about the effectiveness of preprocessing steps, which are critical in natural language tasks.
o Table 4 (Gemini’s Evaluations):The scores presented (e.g., P1 = 8.2, P2 = 3.9) lack contextual analysis or a clear explanation of why Gemini rated some prompts poorly (e.g., P2 and P4).
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.