Systematic evaluation of ChatGPT performance in providing renewable energy information

Introduction

The evolution of natural language processing (NLP) tools led to a new era of information discovery. Traditional methods of defining words in contexts have advanced from basic statistics to machine learning methods. A word in text is now defined by a dense vector that represents not only the word and its accompanying words, but also its position, its semantics, its syntax, and many more. The new era of machine learning and NLP word embedding models is called large language models (LLMs). These models adopt all the concepts of NLP in addition to vectorization, with more information about the words and their accompanying ones. Further to this, they define word positions and add masks to the vector to find the best word representation. GPT3, which is short for Generative Pre-trained Transformer, and its advancement, such as GPT3.5 and recently GPT4, made a big change in this field. These models were scaled up to process very large texts, leveraging the concepts of transformers. The transformers are trained on a parameter count of 175 billion (Brown et al., 2020). The Word embeddings generated by the transformers are fine-tuned for the task of building ChatGPT.

ChatGPT is a chatbot built by OpenAI using one of the large language models, namely GPT-3.5. It has proven its ability to deliver interesting information in a wide range of fields. Since the time ChatGPT was presented, many applications have revolutionized, such as customer support, question answering, language translation, and many more (Ray, 2023; King & ChatGPT, 2023; Wu et al., 2023; Biswas, 2023; Marr, 2023; Bukar et al., 2024b; Bouteraa et al., 2024).

Figure 1 shows the major steps of ChatGPT’s main concept of learning, which is known as Reinforcement Learning from Human Feedback (RLHF) (Trivedi et al., 2023). In the first step of Fig. 1, the human is tuning the prompt learning progress by proposing the prompt and their ideal response. Then, in step 2, the Reward Model (RM) is used by a human labeler to rank different responses from the best to the worst. The model accordingly learns the quality of the responses depending on human preferences. The Proximal Policy Optimization (PPO) technique is applied in step 3 to optimize the model based on RM ratings, which in turn enables the model to produce higher-quality and more appropriate responses according to the prior preferences. Since its launch, ChatGPT has attracted nearly 100 million registered users (Milmo, 2023).

ChatGPT has attracted nearly 100 million registered users since its launch in 2021. SimilarWeb analytics has also shown that the ChatGPT website received more than 1.5 billion visits with a bounce rate of 39% (SimilarWeb, 2023). Further to this, several studies have highlighted that ChatGPT’s ability to provide proper information was satisfying to human requirements in different fields (George & George, 2023; Dwivedi et al., 2023; Johnson et al., 2023; Bommineni et al., 2023; Lund & Wang, 2023). Nevertheless, concerns have arisen about the ethical considerations and regulatory frameworks needed to govern ChatGPT as an information tool (Bukar et al., 2024a). In fact, ChatGPT is more than just a conventional chatbot; rather, it is able to act in different contexts like a real person. For example, ChatGPT can be asked about how to solve mathematical problems or programming exercises. Furthermore, it can chat interactively, exchanging responses back and forth. It is also able to chat in a human-like manner, retrieving information and answering questions.

The future of linguistic communication may change after this tool. In the near future, we may speak to people thinking of them as humans, but they are human-like robots (Hubots). Having someone to talk to and ask questions is the essence of any communication. However, this is still a chatbot that is trained on human-generated data. Hence, a multitude of questions arise regarding the functionality and reliability of the ChatGPT chatbot. These questions include assessments of its accuracy (e.g., How accurate is the information provided by ChatGPT compared to human expert knowledge?), considerations regarding the adequacy of information provision, and reflections on its potential (e.g., Can ChatGPT substitute experts?). Additionally, concerns are raised regarding the sourcing and validation of data utilized in training such chatbots, particularly within specialized domains that directly impact human lifestyles (e.g., Can ChatGPT aid in the advancement of renewable energy technologies?). This research is represented as the base framework for investigating of ChatGPT’s capability in providing related and represented information in specialized domain like renewable energy.

Renewable energy has been selected as the focal point of our case study to explore LLMs such as ChatGPT, primarily because of its significant contribution to the reduction of CO₂ emissions (Caglar, 2020), which is a critical global concern in the present era of climate change. Renewable energy is a source of energy that generates power naturally with little or no emissions (Owusu & Asumadu-Sarkodie, 2016). It is characterized by utilizing its natural way of energy generation with minimal or no emissions. Reduction of emissions plays an important role in mitigating environmental concerns, particularly reduction of CO₂ emissions, which are central to reducing climate change (Raihan et al., 2024; Raihan & Bari, 2024). Applying LLMs and artificial intelligence (AI) technologies in renewable energy holds the ability to improve the efficiency of renewable energy projects using streamlined processes, providing recommendations for resource allocation, and helping innovative solutions (Lukianykhin et al., 2024). In predictive modeling and optimization methods for intelligent control and monitoring systems, these methods give a range of tools and approaches to improve the performance and reliability of renewable energy systems while utilizing AI and LLM. In addition to that, renewable energy can benefit from LLMs through training them on its data and getting insights that can help in achieving accurate decisions (Rane, 2023; Atwa, Almazroi & Ayub, 2024).

ChatGPT’s task in this study is to understand the formulation of questions and information regarding renewable energy sources and provide responses. The questions that follow and answers have been collected from a variety of reliable renewable energy sources. Then, we systematically compared responses from ChatGPT and domain experts, applying text analysis as well as hypothesis testing using a paired sample t-test to evaluate the degree of similarity. We aim to systematically evaluate the accuracy, depth, and reliability of ChatGPT-generated information in the renewable energy domain, with the goal of clarifying its potential benefits and risks as a supportive tool. Specifically, we investigate the extent to which GPT responses align with human expert answers in both lexical and semantic dimensions, and further evaluate the responses using an independent LLM-based assessor, namely, Gemini. It is imperative to mention that this study focuses on general, non-technical renewable energy information that is intended for public understanding and does not assess expert-level or engineering-specific content. The contributions of this research can be summarized as follows:

• 1.

  Contributes to understanding the capabilities and limitations of ChatGPT in the domain of renewable energy. By systematically comparing ChatGPT responses with those of domain experts, insights into the accuracy and comprehensiveness of ChatGPT responses are garnered, shedding light on its potential utility as an information tool within this specialized field.

• 2.

  The research methodology employed, particularly the systematic comparison using text analysis techniques, provides a replicable framework for evaluating the performance of ChatGPT as an LLM in a specialized field. This methodological approach can enhance the rigor and standardization of evaluating LLM performance.

The remainder of this article is organized as follows: background and related work are presented in the next section. After that, the research methodology and the proposed systematic comparison framework using text analysis techniques are explained in detail. Then, the results and their discussions are presented, and finally, the conclusions of the research and future prospective work are provided.

Related works

Large language models have revolutionized NLP and AI, offering unprecedented capabilities in understanding and generating human-like text. Their potential applications are vast and span multiple domains (De Angelis et al., 2023). LLMs build on a new advancement in the field of artificial neural networks called transformers. Transformers were first introduced by Vaswani et al. (2017) in their research article entitled “Attention is All You Need”. They have stated that transformers could achieve state-of-the-art results on a variety of natural language processing tasks, including machine translation, text summarizing, and question answering. Traditional word embedding models, like Word2Vec (Mikolov et al., 2013), GloVe (Pennington, Socher & Manning, 2014), Fasttext (Bojanowski et al., 2017), and their variants have been applied in various fields such as translation (Wolf et al., 2014; Sitender, Sushma & Sharma, 2023; Gogineni, Suryanarayana & Surendran, 2020), named entity recognition (Arguello-Casteleiro et al., 2021; Sienčnik, 2015; Kim, Hong & Kim, 2019; Hou, Saad & Omar, 2024), healthcare (Ibrahim et al., 2021; Craven, Bodenreider & Tse, 2005; Hasan & Farri, 2019; Amunategui, Markwell & Rozenfeld, 2015; Zhu et al., 2016; Hajim et al., 2024), and many others. Despite many improvements to these models and their applications, they have not reached the level of the new NLP and LLM advancements. LLMs and their variants have revolutionized communication, decision making, critical thinking, and data analysis. ChatGPT is a notable example of the innovations stemming from the advancements in LLMs.

Significant research has been undertaken since the launch of ChatGPT. While Rathore has employed ChatGPT to improve textile production by studying the customer’s preferences to provide relevant recommendations. Rathore, however, has yet to justify how ChatGPT works as a recommender system (Mollman, 2023). Bommineni et al. (2023) have examined ChatGPT with the Medical College Admission Test (MCAT), and the results were promising. ChatGPT was able to answer over the median of approximately $277K$ students. Besides that, ChatGPT was able to provide insightful responses that were nearly identical to the standard responses. Such an ability opens the door for creating additional questions and/or insights to help medical students when it comes to MCAT (Bommineni et al., 2023). A similar study by Kung et al. (2023) examined ChatGPT, but this time on the United States Medical Licensing Exam (USMLE). Their study has reported that ChatGPT showed a surprising performance and overcame the USMLE.

Jiao et al. (2023) have evaluated ChatGPT on translation tasks using different language datasets ranging from well-known languages to distant languages. While ChatGPT was a competitor to popular translators, such as Google Translate, for popular languages, it was less effective for distant languages. Further to this, ChatGPT has been tested on 20 mainstream NLP datasets representing seven evocative tasks by Qin et al. (2023). Their experimental study concluded that ChatGPT was able to achieve several reasoning tasks, such as arithmetic operations, although it has performed some tasks imperfectly. On the other hand, Li, Jiang & Mostafavi (2023) have conducted an empirical assessment of ChatGPT responses concerning protective action information. The responses were collected from ChatGPT, and then a survey was developed using these responses. A total of 38 specialized experts in the field of emergency response have assessed the survey and confirmed that the ChatGPT responses were accurate and relevant to the subject area.

Another study has evaluated ChatGPT on different questions related to the medical license exam in the United States using four different datasets. The results demonstrated that ChatGPT outperforms InstructGPT by 8.15% on average. The study also recommended that ChatGPT can be used as a medical tutor (Gilson et al., 2023). A more relevant study was done by Johnson et al. (2023), focusing mainly on the medical field, in which around 280 questions were collected from 33 medical doctors from the same academic center. This may be an indication of bias in the data; therefore, we have gathered our data sample from various online sources, where several experts answered questions.

In the same direction, a study has investigated the reliability of $ChatGPT$–4 in providing medical information to patients and healthcare professionals. The Ensuring Quality Information for Patients (EQIP) tool has been applied to measure the quality of information for five different diseases. Human in the loop approach has been applied for question/answer in an independent setting. For all 36 items of the EQIP, the median score was 16 $(IQR14.5$–18). There was 60% agreement between the guidelines and ChatGPT responses (15/25). Significant agreement was noted using Fleiss’ Kappa measure, which estimated the inter-rater agreement at 0.78 with $P<0.001$. Overall, the responses that ChatGPT delivered were 100% consistent compared to online medical information (Walker et al., 2023).

The evaluation of ChatGPT responses in the field of renewable energy remains an underexplored area. Although a handful of articles and blogs touch on related themes, they generally lack solid experimental design and fail to incorporate either quantitative or qualitative analysis that would allow for a reliable assessment of ChatGPT’s informational accuracy and consistency. ChatGPT has been investigated in published research for its potential in intelligent energy management applications, like forecasting energy usage and giving consumers energy-saving guidance (Shah, 2023). However, these studies do not evaluate the trustworthiness or depth of the information provided, nor do they compare ChatGPT’s responses against expert-level standards, which is critical when addressing complex scientific domains such as renewable energy. Another study has introduced an Energy service chatbot that combines the power of LLMs and a multi-source Retrieval Augmented Generation (RAG) framework (Arslan, Mahdjoubi & Munawar, 2024). This Energy service chatbot has been evaluated against LLMs like Llama2, Llama3, and Llama3.1 models, demonstrating improvements in precision, recall, accuracy, and F1-scores with RAG integration. While it benchmarked against models like LLaMA2 and LLaMA3.1, there was limited discussion on how the chatbot’s responses align with real-world expert understanding of renewable energy systems.

RE-LLaMA has been presented as a domain-specific LLM model for renewable energy and built based on LLaMA 3.1 8B (Gabber & Hemied, 2024). RE-LLaMA has been fine-tuned using nearly 1,500 academic articles, covering a wide range of topics, including optimization, hydrogen technologies, and the design of renewable systems, along with decarbonization strategies. A significant amount of work has been dedicated to data processing and cleaning. RE-LLaMA abilities have been evaluated with zero-shot and few-shot learning. While RE-LLaMA narrows the domain gap, the evaluation remains centered on technical model performance rather than end-user informational reliability or expert alignment. Additionally, no human-in-the-loop assessment was conducted to benchmark responses against subject matter expertise.

In contrast to prior investigations, our research uses a structured comparison analysis with subject matter experts to directly address the significant gap in assessing ChatGPT’s responses on renewable energy subjects. Rather than depending exclusively on task-specific performance measures or standards, we use a methodical evaluation that compares ChatGPT’s responses to experts’ responses in order to determine how accurate, thorough, and pertinent they are. This human-grounded comparison approach provides unique insights into the model’s capabilities and limitations when applied in high-stakes, knowledge-intensive domains.

Materials and Methods

In this section, we discuss the process of data collection and data analysis approaches. The proposed methodology discusses response similarity approaches between human and ChatGPT-generated responses. This includes details of both lexical and semantic measurements. Our research provides a framework for evaluating ChatGPT’s accuracy in providing renewable energy information.

Similarity measurements

In this section, we comprehensively evaluate response similarity between human and ChatGPT responses, employing both lexical and semantic measurements. Initially, for lexical measurements, we used text similarity metrics, such as Jaccard, Cosine, Levenshtein, and Sorensen-Dice to quantify similarity. These metrics have the ability to measure the extent of term overlap between two texts; thus, they can capture the surface-level similarity between the ChatGPT responses and human expert responses. This process is then followed by a semantic analysis using Google’s Word2Vec embeddings to capture contextual similarity. This textual similarity and semantic metrics were driven by the need to evaluate ChatGPT responses in a consistent manner with how humans process and understand language. Moreover, an evaluation using an LLM competitor, Gemini, was conducted to measure the degree of similarity between ChatGPT and a human expert. This section provides a comprehensive evaluation of ChatGPT responses related to the renewable energy domain.

The first approach, lexical measurement, involved response/prompts matching to assess whether ChatGPT responses $B_i$ share greater similarity with the reference than a human expert’s $A_i$. In this context, we have computed for each prompt $i$ a paired difference, which involves finding the common words that $B_i$ and $A_i$ have shared as $D_i=B_i-A_i$. The similarity scores of $B_i$ and $A_i$ were respectively measured by conventional text similarity metrics such as Jaccard, Cosine, Levenshtein, or Srensen˘Dice. We then performed a one-sample, one tailed paired t˘test with $H_0:{\mu }_D=0$ vs $H_a:{\mu }_D>0$. With 95% confidence interval for the mean difference ${\mu }_D$, which is given by $\overline{D}\pm t_{0.975,n-1}\frac{s_D}{\sqrt{n}}$, where $s_D$ denotes the standard deviation and $n$ is our sample size. To keep it simple and straightforward, we are using A as a reference to human expert responses and B for ChatGPT responses for the rest of this article.

The detailed steps of this approach are presented in Algorithm 1. The processes begin by defining the input and required parameters, followed by pre-processing to remove unnecessary contents (e.g., stopwords and punctuation) to start response tokenization and find common words as per Algorithm 1, Lines 1–5. The process of finding common words between two sets of responses representing a human expert response A and a ChatGPT response B involves several steps: ( $i$) removing stopwords and punctuation, ensuring that only meaningful words are considered, ( $ii$) the text is tokenized, breaking it down into individual words or tokens by splitting the text based on whitespace characters (spaces, tabs, etc.), ( $iii$) These tokens are then converted into sets, data structures that contain unique elements, to eliminate duplicate words, ( $iv$) Finally, the intersection or operation is then applied between the two sets, resulting in a new set containing only the words that are common to both the human expert response and the ChatGPT response. In this final step, the Jaccard (J), Cosine (C), Levenshtein (L), and Sorensen-Dice ( $S_D$) were used as similarity metrics in Algorithm 1, lines 6–10. The mathematical formulas and corresponding functions for the adopted similarity metrics are presented in Algorithm 1, Lines 24–39.

Algorithm 1:

Responses matching process with text similarity metrics.

Input: Expert ( $Respons{e}_{A_{list}}$ = $\{A_{i_1},\dots ,A_{i_x}\}$); ChatGPT ( $Respons{e}_{B_{list}}$ = $\{B_{i_1},\dots ,B_{i_x}\}$);

1 Jaccard (J); Cosine (C); Levenshtein (L); SorensenDice (SD)

Result: $Prompt{s}_{\mathrm{L}\mathrm{i}\mathrm{s}\mathrm{t}}=\left\{CommonWord{s}_{i_x},J_{i_x},C_{i_x},L_{i_x},S_{D_{i_x}}\right\}$;

procedure MatchResponses $(A_{i_1,\dots ,i_x}$, $B_{i_1,\dots ,i_x})$

Let $A_{i_1}$ be the set of words in Response A for Question 1 (human expert response)

Let $B_{i_1}$ be the set of words in Response B for Question 1 (ChatGPT Response)

Remove stopwords and punctuation from A and B

$CommonWords\leftarrow A\cap B$ ▹ Find common words

$J\leftarrow$ Jaccard(A, B)

$C\leftarrow$ Cosine(A, B)

$L\leftarrow$ Levenshtein(A, B)

$S_D\leftarrow$ SorensenDice(A, B)

return $CommonWord{s}_{i_x},J_{i_x},C_{i_x},L_{i_x},S_{D_{i_x}}$

end procedure

procedure MatchAfterStemming( $A_{i_1,\dots ,i_x}$, $B_{i_1,\dots ,i_x}$)

Apply SnowballStemming function to both HumanResponse and ChatGPTResponse;

Compute word similarity between stemmed responses;

Apply Procedure 1;

return $CommonWord{s}_{i_x},J_{i_x},C_{i_x},L_{i_x},S_{D_{i_x}}$

end function

function SnowballStemming(text)

Load Snowball stemmer

Tokenize text into words: $text=\{w_1,w_2,\dots ,w_n\}$

for each word wi in the text) do

Apply stemming to word wi: $stem(w_i)$

Return: Stemmed text

end function

function Jaccard(A, B)

$J(A,B)=\frac{|A\cap B|}{|A\cup B|}$

return J(A, B)

end function

function Cosine(A, B)

$C(A,B)=\frac{A\cdot B}{\Vert A\Vert \cdot \Vert B\Vert }$

return C(A, B)

end function

function Levenshtein(A, B)

Compute L distance between A and B

return L distance

end function

function DorenseDice(A, B)

$S_D(A,B)=\frac{2|A\cap B|}{|A|+|B|}$

return $S_D(A,B)$

end function

DOI: 10.7717/peerj-cs.3295/table-5

Moreover, as part of the first approach, we employed the Snowball stemming tool (Porter, 2001) to convert the words from both responses into their basic forms, thereby enhancing the degree of similarity marginally. The procedure named ‘MatchresponsesAfterSnowballStemming’ in Algorithm 1 involves several steps to compare responses from both humans and ChatGPT after applying Snowball stemming. Initially, the ‘SnowballStemming’ function is applied to both the HumanResponse and ChatGPTResponse, converting them into their stemmed forms (aka, converting text into basic forms) to improve similarity measurement. The ‘SnowballStemming’ function process involves loading the Snowball stemmer and tokenizing the input text into words. Then, it iterates over each word in the text and applies Snowball stemming to convert it into its basic form.

Subsequently, the word similarity between the stemmed responses is computed. These stemmed responses passed to the ‘MatchResponses’ procedure in Algorithm 1 to apply the similarity metrics measures in Algorithm 1, lines 6–10. Finally, the procedure returns the common words and similarity scores, including J, C, L, and $S_D$, providing insight into the degree of similarity between human and ChatGPT responses after applying Snowball stemming. It is important to note that while an increase in similarity of words has occurred, it does not indicate or prove that ChatGPT has produced a similar/accurate response. Hence, we adopted the semantic measures in the second approach.

In the second approach, which was semantic measurement, we realized that evaluating ChatGPT’s effectiveness based solely on word similarity is deemed insufficient. Therefore, to address this limitation, we incorporated another measurement focusing on semantic similarity in the second approach. For this purpose, we utilized Google’s pretrained embeddings, consisting of three billion embeddings derived from the Google News Corpus and trained using the Word2vec Model. These embeddings enabled the comparison of semantic similarities between human responses and those generated by ChatGPT. Algorithm 2 provides a comprehensive approach to evaluate response similarity, incorporating semantic-level measurement similarity. The algorithm begins with the ‘SemanticSimilarity’ procedure, which takes two inputs: ExpertResponse and ChatGPTResponse, representing the human and ChatGPT responses, respectively. Following this, the procedure computes the semantic similarity score between the human and ChatGPT responses (Algorithm 2, lines 1, 8). Finally, the algorithm evaluates the ‘SemanticSimilarity’ function and calculates the semantic similarity between two responses, A and B, using cosine similarity. It starts by loading Google’s pretrained Word2Vec embeddings. Then, it represents the responses as word vectors and computes the cosine similarity score between them. This score reflects the semantic similarity between the responses, capturing their contextual similarity beyond the simple word matching adopted in the previous approach. Finally, the function returns the semantic similarity score $S{S}_c$.

Algorithm 2:

Semantic similarity evaluation using Google Word2Vec.

Input: Expert ( $Respons{e}_{A_{list}}$ = $\{A_{i_1},\dots ,A_{i_x}\}$); ChatGPT ( $Respons{e}_{B_{list}}$ = $\{B_{i_1},\dots ,B_{i_x}\}$);

Result: SemanticSimilarityScore (SSc);

1:  procedure SemanticSimilarity ( $A_{i_1,\dots ,i_x}$, $B_{i_1,\dots ,i_x}$)

2:   Input: ExpertResponse A, ChatGPTResponse B

3:   Output: Semantic similarity score S

4:   Load Google’s pretrained Word2vec embeddings

5:   Represent A and B as word vectors: $A=\{a_1,a_2,\dots ,a_n\}$, $B=\{b_1,b_2,\dots ,b_m\}$

6:   Compute semantic similarity score SSc between A and B using cosine similarity:

7:     $S{S}_c=\frac{{\sum }_{i=1}^n{\sum }_{j=1}^m{a}_i\cdot b_j}{\sqrt{{\sum }_{i=1}^n{a}_i^2}\cdot \sqrt{{\sum }_{j=1}^m{b}_j^2}}$

8:   Return: Semantic similarity score SSc

9:  end procedure

DOI: 10.7717/peerj-cs.3295/table-6

The third approach is to further examine the semantic similarity of ChatGPT responses. An additional assessment was conducted through a comparative analysis with responses generated by Gemini, which is another prominent chatbot. Gemini was not used as a comparative benchmark in this study; rather, it served merely as an additional layer for the similarity measure between the ChatGPT and human expert responses. Gemini, formerly known as Bard, is a widely prominent chatbot powered by the Pathways Language Model (PaLM) (McIntosh et al., 2023), developed by Google in February 2023. Leveraging the capabilities of Gemini, it provides a benchmark for evaluating ChatGPT responses’ accuracy in comparison to human expert knowledge. Gemini has been selected as an evaluation tool because it has been configured, trained, and tested differently from ChatGPT. Also, Gemini is a key competitor and has big potential to reshape the AI landscape. In this context, Gemini was given a set of prompts, and similarity scores were collected for subsequent analysis/observation. As shown in Fig. 2, the ground truth dataset of prompts, along with human expert responses and ChatGPT responses, was presented to Gemini for similarity measure and scoring.

Results and discussion

In our comprehensive evaluation, a mixture of qualitative measurements has been applied to evaluate the performance. In the quantitative part, traditional measures of similarity have been employed. Initially, a direct similarity was considered, and then a semantic similarity. Further to this, both the human expert and ChatGPT responses have been presented to another popular chatbot to confirm the ability of ChatGPT to generate accurate responses and increase the power of the qualitative evaluation. This allowed us to assess the similarity of the two responses. The multifaceted approach allowed us to evaluate the accuracy and relevance of the information provided by ChatGPT.

In order to assess the ChatGPT responses regarding renewable energy prompts, a set of similarity matrices was applied. This evaluation compares a group of words generated by human experts to those produced by ChatGPT using Jaccard similarity, Sorensen dice, Levenshtein, and Cosine Similarity metrics. Initially, a word-to-word comparison has been conducted, then all punctuation and stop-words have been dropped, and words have been stemmed to their basic form. Using a paired sample t-test, there was insufficient evidence to reject the null hypothesis $p>0.05$, which is clearly due to the poor word-to-word comparison approach being applied in the first phase of this experiment. The small effect size $Cohe{n}^{\prime }sdz=0.08$ further supports the lack of practical significance between the two sets of answers in this approach. Table 2 presents the pre- and post-processing mean score of these similarity matrices.

It can be observed from Table 2, that the similarity scores were underwhelming, with cosine similarity peaking at only about 60 and 61 for pre- and post-processing, respectively, which was the highest score recorded among all metrics. In fact, pre- and post-processing similarity scores show only slight changes due to shortened responses of ChatGPT and human experts, which are characterized by limited stop-words and punctuations. It is imperative to mention that this similarity alone does not definitively determine whether ChatGPT responses are closely aligned with those produced by human experts. One effective technique is to use semantic similarity, which captures the overall meaning of each response. In order to explore the semantics similarity in greater detail, we conducted a thorough assessment using Google News vectors. This was followed by the cosine similarity measure because it produced the most promising results during the syntactic similarity evaluation.

Figure 3 demonstrates the mean semantic similarity score measured by Google News vectors along with cosine similarity and presented using Kernel Density Estimation. The mean semantic similarity was about 87% across all 52 prompts. It is obvious that the most semantic similarity values were clustered around 91%. Such clustering indicates not only a high central tendency but also low variance, confirming that ChatGPT’s responses closely mimic those of a human expert. Further to this, a second round of paired sample t-test has been applied, and this time we have successfully rejected $H_0$ in favor of $H_a$ with a $p<0.05$, emphasizing the importance of the semantic similarity measure and that ChatGPT responses were statistically significantly more relevant than those of the human experts when it comes to renewable energy prompts.

Further to this, an additional assessment has been conducted to evaluate the accuracy of ChatGPT responses. This has been done by comparing ChatGPT responses with expert ones using Gemini (named before as Bard). Gemini is a popular chatbot driven by the Pathways Language Model (PaLM) (McIntosh et al., 2023) and developed by Google in February 2023. For this purpose, a 10-prompt questionnaire was designed and presented to Gemini, and then the responses were collected for analysis. Table 3 demonstrates prompts and their scales of responses that were presented to Gemini. These prompts include commonly asked renewable energy questions that were selected from our dataset, along with the responses provided by an expert human (response A) and those generated by ChatGPT (response B). Then, Gemini was asked to assess the similarity between responses A and B using a 1–10 scale, to determine if the responses share the same viewpoints and to decide which option is the most relevant. The average similarity score by Gemini, for each question, is presented in Table 4.

The mean response scores presented in Table 4 for prompts 1–8 indicate a high degree of similarity in responses between human experts and ChatGPT. In the ground-truth dataset, prompts P2 and P4 led to very short responses from human experts, while ChatGPT has provided much more extensive replies. Further to this, these prompts assess the extent to which responses align in perspective, and since human perspectives often diverge, leading to varying levels of agreement (Surdek, 2016; Johnson, 2019). Therefore, Gemini scores refer to the fact that ChatGPT’s inferences on P2 and P4 were much clearer than those of the human experts. Gemini has indicated that the mean response score of the eight prompts is approximately 4.2 out of 5, reflecting how close response A was to response B. The evaluations of the message content (P3) and the topic argument (P5) were all above 4, as were the terminology (P6) and recommendations (P7). Prompt (P6), which measured the degree of terminological similarity, was 4.5 out of 5, reflecting a considerable alignment of responses. Overall, the outcomes of our semantic similarity analysis are robust and highly consistent, confirming the reliability of our methodology.

Figure 4 shows the distribution of the scores for these 8-Prompts questionnaires for the 52 questions. As we can observe from this figure, most of the scores fall within the range of 4 to 5 scales, indicating high correlations and text representations of the ChatGPT responses to the human responses. Furthermore, P9 and P10 were included for further verification of the accuracy of the ChatGPT responses.

Figure 5 represents each response in three categories, i.e., “A”, “B”, and “Both”, with 95% confidence intervals that were calculated using the Wilson score interval. For P9, Fig. 5A, Gemini reported that response B, which is the ChatGPT response, was more related for 73% of the prompts. The rest of the responses were shared between both A and B. For P10, Fig. 5B, Gemini reported that ChatGPT responses represented the prompts more than human responses. This was observed by 76% of replays, which reported that B was the best, only 1% referring to A as the best, and 23% showed that both A and B represented the prompts. Since ChatGPT has contributed an additional 23% similarity across the “Both” category, this would get ChatGPT’s total representation of the responses to 99%, representing the prompts. This clearly indicates that most of ChatGPT’s responses match or outperform those of the human experts. This experiment demonstrates that the renewable energy information provided by ChatGPT was relevant and at the point, making ChatGPT a promising tool that can help improve the user experience when it comes to the renewable energy space, presenting information and supporting user inferences.

Although this article acknowledges ethical issues such as the potential misinformation or ChatGPT’s general training bias, we ensured any bias or misinformation was avoided by benchmarking against a manually crafted, domain-specific “ground truth” dataset sourced from authoritative renewable-energy materials. In other words, rather than comparing ChatGPT responses to an arbitrary reference, we matched them against a curated set of expert responses known to be as free as possible from common online noise or superficial framing.

Overall, OpenAI continues evolving and has released almost six versions of its GPT algorithm, starting with its first version in 2018 up to the most recent one, GPT-4 Turbo. Recognizing the critical importance of renewable energy early on, which was further emphasized through this study, ChatGPT now provides an informative guide on renewable-energy management, directly supporting one of the United Nations’ key sustainable development goals on the following link: https://chatgpt.com/g/g-WvMVuNHoF-renewable-energy. It is impressive to mention that with the evolution of ChatGPT, periodic evaluations appear feasible. This study provides quantitative measures and can serve as a baseline for potential longitudinal studies tracking ChatGPT’s progress over time.

Providing end users with reliable, carefully selected information on renewable energy using ChatGPT can fundamentally transform how individuals engage with sustainable technologies. When individuals are provided with clear, reliable responses to their prompts, they are more likely to adopt evidence-based best practices, such as optimizing solar panel orientation, adjusting load schedules to accommodate off-peak generation, or practical tips for demand-side management. This not only improves overall energy efficiency but also builds trust in renewable solutions, enabling end users to see themselves as active participants rather than passive observers. While ChatGPT has the potential to enhance public engagement with renewable energy, it is important to recognize its limitations, particularly its tendency to hallucinate, or generate information that appears accurate but is factually incorrect or misleading. Such inaccuracies may undermine user confidence and hinder the adoption of sustainable practices if not identified and corrected. Therefore, ChatGPT should be viewed as a supportive, not authoritative, tool, most effective when its outputs are verified through up-to-date, expert, and location-specific sources.

Conclusions

A new era of information generating has emerged with the disclosing of LLMs and their recent technology of conversational chatbots such as ChatGPT. This study aimed to compare the information generated through ChatGPT to human experts with a focus on renewable energy, an important domain for future sustainable development. Our research compiled 63 prompts generated by humans and the corresponding responses from various renewable energy platforms. A head to head comparison was conducted to evaluate the relevance and validity of ChatGPT responses against those of human experts using lexical and semantic measurements. Further to this, Gemini has been asked to measure the similarity between ChatGPT responses and those of human experts, proving that ChatGPT is an impressive source of information, particularly when it comes to renewable energy, with almost 99% of information representation accuracy. ChatGPT has provided insights into renewable energy across diverse aspects that complement and, in some areas, extend the human expert’s knowledge. While we recognize that larger-scale benchmarks are important for broad statistical generalization in this context, expanding to a more extensive set of prompts is therefore planned as a second phase, allowing future work to validate and extend these initial findings across a more diverse and robust dataset. Lastly, we emphasize that AI tools depend on humans and are not a substitute for them. Rather, humans can even leverage AI to assist with tasks that require intensive human effort or high computational skills.

Limitation

There may be some possible limitations that can be addressed in future work. While Word2Vec embeddings are a popular method for semantic similarity analysis, they may inherit biases from their original training sample. Moreover, incorporating more recent transformer-based models like BERT could lead to richer contextual representations. On the other hand, our handcrafted dataset appears relatively small to examine LLM-based chatbots like ChatGPT’s performance on renewable energy. Although this has been acknowledged, this study primarily serves as a proof-of-concept, and thus, larger, more diverse, and technical prompts will be required to confirm and extend our findings at expert-level or engineering-specific content.

Supplemental Information