G-Pro bot: a GraphRAG-powered GIS (Geographical information system) chatbot

G-Pro bot development

In this section, first, we construct a GIS knowledge base, which includes introductory books, professional books and software operation manuals. Second, we chunk the text for the subsequent steps. We then extract key information, such as entities, relationships, and attributes, from the chunked text. On the basis of this extracted information, we build a knowledge graph (KG)—a structured representation of knowledge using nodes for entities and edges for relationships—and apply the Leiden algorithm to partition the graph into communities. With the LLM, we generate a hierarchical community abstraction. After a query is received, the answer is generated via the abstraction created in the previous step. The development of the G-Pro Bot is completed through the steps outlined above. The overall construction process is based on Microsoft’s GraphRAG project (Microsoft, 2024), which leverages knowledge graphs to structure information, enabling a more comprehensive and precise retrieval of interconnected facts and entities for augmented generation. The development workflow of the G-Pro Bot is shown in Fig. 1.

LLM integration and enhancement

The state-of-the-art LLM, Llama3.1-8B (Grattafiori et al., 2024), and the embedding model, Nomic-Embed-Text (Nussbaum et al., 2024), were selected to be integrated and improved in conjunction with GraphRAG. Llama3.1-8B is widely used for pretraining tasks because of its multilingual support and outstanding performance. Additionally, its context length of 128K enables it to handle long texts effectively, making it particularly well suited for G-Pro Bot’s knowledge integration and generation tasks. Nomic-Embed-Text, one of the most influential text embedding models, achieves excellent performance in processing both short and long text tasks.

Our selection of Llama3.1-8B was guided primarily by a balance between its demonstrated strong performance in various natural language understanding tasks and its computational efficiency and accessibility. While there are larger parameter models (e.g., 70B parameters or more) that provide impressive capabilities, their deployment typically requires substantial computational resources and entail considerably longer initialization and inference times. By focusing on an 8B parameter model, our study demonstrates the efficacy of the GraphRAG approach within a more resource-accessible framework, making our findings more readily transferable and reproducible for researchers with conventional computational setups. We acknowledge that exploring the performance gains of GraphRAG when they are integrated into even larger foundational models is a valuable direction for future research.

The GraphRAG implementation process was conducted on a computer equipped with an NVIDIA Quadro RTX 8000 GPU with 48 GB of memory.

Together, GraphRAG, Llama3.1-8B, and Nomic-Embed-Text collaborate to facilitate knowledge integration and response generation for G-Pro Bot, as outlined in the process below. To perform the same setup and implementation, please download the project on Zenodo at https://zenodo.org/records/16730421.

• 1.

  Source document split: The corpus is divided into text chunks via GraphRAG, which serve as the basis for extracting the knowledge graph. Overlaps between chunks are managed to ensure semantic continuity.

• 2.

  Element instances extraction: GraphRAG is employed to identify extraction instances from the text chunks, focusing first on entities (e.g., names, types) and then on relationships between entities and their attributes. These extracted instances are managed as the basis for constructing the knowledge graph.

  The core extraction methodology for GraphRAG that we used is LLM prompting. Our chosen LLM, Llama3.1-8B, is used for this purpose. For entity extraction, the LLM is prompted with specifically crafted instructions to identify named entities and generate their descriptive attributes from each text chunk. Simultaneously, for relationship extraction, the LLM is prompted to describe the connections and their types between identified entity pairs within each text unit. The specific prompts used for these initial extractions are detailed in Fig. 2. In this generative process, the LLM directly outputs the structured entities and relationships on the basis of its understanding of the provided text and prompt.

  To address the challenge of missing instances, GraphRAG implements an iterative and self-correcting extraction mechanism. After an initial extraction pass, the system can use specific prompts to guide Llama3.1-8B to reevaluate and iteratively extract any remaining missing entities or relationships. The specific prompts used for this iterative re-evaluation and extraction process are also presented in Fig. 3. This multipass approach significantly improves the recall of the extraction process.

• 3.

  KG construction: GraphRAG generates a preliminary knowledge graph via the extracted entities and relationships, with entities represented as nodes and relationships as edges.

• 4.

  KG partition: The KG is partitioned into communities using a community detection algorithm, which determines the strength of the relationships between nodes. Specifically, we use the Leiden algorithm, which excels at recovering the hierarchical community structure of large-scale graphs. Each community corresponds to a set of semantically related entities and relationships and is mutually exclusive. The hierarchical nature of the community divisions ensures that the G-Pro Bot can provide fine-grained answers ranging from a global scope to a local scope.

• 5.

  Community summaries generation: GraphRAG defines the hierarchy of summary generation, from overviews to communities, and manages the semantic information for each community. Llama3.1-8B generates natural language summaries on the basis of the extracted entities, relationships, and community content, ensuring that the summaries are both comprehensive and fluent.

• 6.

  Answer generation: With hierarchical community summaries, the answer generation process is conducted in two phases. The first phase, known as the map phase, involves Llama3.1-8B, which generates localized answers on the basis of community summaries relevant to the query. To increase the relevance of the retrieval results, Nomic-Embed-Text is employed to generate vector embeddings for entities, relationships, and community summaries. These embeddings enable the identification of the most relevant communities or entities for a given query on the basis of vector similarity, increasing the accuracy of information retrieval. The second phase, the reduce phase, integrates the localized answers into a comprehensive global response, with Llama3.1-8B refining the linguistic style to align with the user query.

G-Pro bot evaluation

After we completed the development of G-Pro Bot, we extracted 1–2 key points from each document in the corpus to serve as questioning material. A total of 22 questions were obtained. To evaluate the performance of GraphRAG in terms of contextualization and considering the close relationship between GIS and software operations, we designed four categories of questions: general questions, specific questions, general questions about software, and specific questions about software. The 22 questions and their respective categories are presented in Table 3.

The questions above were used to query three chatbots, including the G-Pro Bot, and the responses were collected and organized, yielding three datasets with a total of 66 responses for subsequent evaluation. The evaluation process is shown in Fig. 4.

Evaluation methodology

Automated evaluation methodology

Our automated evaluation assesses the semantic similarity between model-generated responses and a set of standard answers. Crucially, these standard answers were generated directly from our processed GIS corpus to ensure factual grounding and comprehensiveness. Therefore, this evaluation is specifically designed to compare the performance of the two models that utilize this corpus: our G-Pro Bot (C-Model) and the LangChainFramedRAG Llama3.1-8B (B-Model). The A-Model (plain Llama3.1-8B) was excluded from this part of the evaluation, as it does not have access to the knowledge base, and comparing it against corpus-derived answers would be inappropriate. On this basis, we analyze the discrepancies between the chatbot-generated responses and the standard answers.

The standard answers used for the automated evaluation were established through a rigorous two-stage process to ensure their accuracy and comprehensiveness. First, initial answers were generated directly from our processed GIS corpus using the ChatGPT-4.0 API. Subsequently, these machine-generated answers underwent a critical review by multiple independent experts holding master’s degrees in GIS from our institute. The experts unanimously confirmed that the answers fully addressed the questions without factual errors, thus validating them as a robust ground truth for our evaluation.

Afterward, we evaluated the responses generated by the two corpus-aware models against the standard answers. To ensure a comprehensive and objective assessment, we employed two automated methods: Sentence-BERT (SBERT) (Reimers & Gurevych, 2019) for its high efficiency and strong performance in measuring semantic similarity (Ha et al., 2021; Joshi et al., 2022), and METEOR (Banerjee & Lavie, 2005) for its accuracy and sensitivity in evaluating text generation quality by incorporating recall and fragmentation penalties.

Human evaluation methodology

As part of the comprehensive human evaluation, a preliminary qualitative review of all the generated responses was conducted by multiple independent experts who hold master’s degrees in GIS from our institution. This review identified general performance patterns and potential error types, including hallucination, omission, irrelevant detail, and lexical redundancy.

To complement the qualitative review and obtain a quantified and diverse set of subjective opinions, we subsequently employed a comprehensive questionnaire. For this evaluation, we collected and assessed the responses generated by all three chatbots: the baseline Llama3.1-8B (A-Model), the LangChainFramedRAG Llama3.1-8B (B-Model), and our G-Pro Bot (C-Model). This questionnaire was administered to a broader pool of users to collect their evaluations of the responses generated by the three chatbots across four dimensions. This systematic approach allowed us to assess the strengths and weaknesses of the chatbots quantitatively from multiple perspectives.

1. Questionnaire design

The questionnaire is structured as follows: (a) Preface: This section introduces the purpose and organization of the questionnaire to the respondents. (b) Personal information: This section collects data on the teaching level and academic background of the respondents, which are used to analyze the evaluations of chatbots from users with different profiles. (c) Chatbot responses: This section presents the responses generated by the three chatbots to various questions. The respondents are asked to carefully read these answers and select the optimal answer on the basis of four specific dimensions. An example of the questionnaire interface is shown in Fig. 5.

The evaluation dimensions are based on the GraphRAG article (Edge et al., 2024) and include four aspects: comprehensiveness, diversity, empowerment, and directness.

Comprehensiveness assesses whether the answer addresses all aspects of the question, adequately addressing the key points. Diversity focuses on whether the answer presents diverse perspectives and insights. Empowerment evaluates whether the answer offers effective guidance or reference for the decision-making or judgment of the user. Directness assesses whether the answer clearly and concisely addresses the question without unnecessary elaboration.

To increase the robustness and generalizability of our results, a manual evaluation of the performance of the G-Pro Bot was conducted on all 22 questions and their corresponding three sets of responses.

2. Respondent selection and bias minimization

Owing to the specialization of the GIS and to validate the questionnaire results, we prioritized participants with backgrounds in geography-related majors. The respondents for this study were faculty members and graduate students from the Institute of International Rivers and Eco-Security at Yunnan University. This selection was based on a convenience sampling approach. The primary rationale for choosing individuals from a specific institute is their specialized knowledge and advanced understanding of a GIS, ensuring that the evaluations of model-generated responses are authoritative and discerning.

To reduce potential biases and ensure the objectivity of the human evaluation, several key measures were implemented:

Blind evaluation: All evaluations were conducted blindly to prevent any preconceived notions or favoritism toward specific chatbot identities. The identities of the three chatbot models were completely anonymized within the questionnaire. In the preface of the questionnaire, the respondents were explicitly informed as follows: ‘Please note that the chatbot identities have been anonymized, and your selections should be based solely on the content of the responses rather than the chatbot’s identity.’

Standardized evaluation criteria: The respondents were given clear and consistent guidelines for assessing responses across the four specified dimensions (comprehensiveness, diversity, empowerment, and directness), as indicated in the preface of the questionnaire. This standardized approach reduced subjective variability among the respondents.

Attention check question: An attention check question was included, and it was GIS related and involved three responses generated by ChatGPT-4 (as shown in Table 4), with increasing levels across the four dimensions. If a participant selected the least comprehensive, informative, and insightful response, their submitted questionnaire was considered invalid. The purpose of this step was to ensure evaluator attentiveness and to filter out careless responses, further mitigating potential bias from inattentive participation.

Table 4:

The attention check question and three responses generated by ChatGPT-4.

The three responses exhibit significant differences across the dimensions of comprehensiveness, richness, empowerment, and directness, with Response B performing the worst. If a respondent selects B, their submission will be considered invalid.

Attention check question	Response A	Response B	Response C
What is Spatial Autocorrelation and its importance to geographical problems?	Spatial autocorrelation refers to the extent to which a phenomenon in a specific area is correlated with phenomena in surrounding areas. Positive spatial autocorrelation indicates that similar phenomena tend to cluster together, while negative spatial autocorrelation suggests that dissimilar phenomena are spatially separated. Its significance in geographic studies lies in its ability to help identify spatial patterns and distributions of phenomena, thereby optimizing resource allocation.	Spatial autocorrelation refers to the spatial correlation between phenomena within a geographic area. It helps us understand whether certain phenomena are distributed in a spatially patterned manner. Understanding this is valuable for the study of geographic issues.	Spatial autocorrelation refers to the correlation between the value of a phenomenon in a geographic space and the values of phenomena in its neighboring areas. Specifically, spatial autocorrelation can be categorized into positive and negative types. Positive spatial autocorrelation indicates that similar values tend to cluster together. For example, in certain areas, regions with high pollution concentrations are often adjacent to other areas with similarly high pollution levels. Negative spatial autocorrelation, on the other hand, implies that dissimilar values are spatially separated, such as different types of land use being distributed more dispersedly in space. Analyzing spatial autocorrelation is crucial for understanding the spatial distribution of geographic phenomena. By studying spatial autocorrelation, researchers can identify and explain the spatial patterns of certain phenomena, which is of significant importance for solving geographic problems. For example, in urban planning, by analyzing the land use patterns of different areas, policymakers can more effectively plan the layout of urban functional zones, ensuring the rational allocation of resources. In the field of public health, spatial autocorrelation analysis can reveal patterns of disease transmission, helping public health authorities implement more precise control measures. Furthermore, spatial autocorrelation analysis has broad applications in various fields such as environmental science, criminology, and socioeconomics. It aids researchers in identifying spatial hotspots or anomalous areas, providing strong data support for decision-making. Through spatial autocorrelation, researchers can not only uncover the spatial patterns of a phenomenon itself but also explore the interactions between different phenomena, revealing potential causal relationships. Therefore, spatial autocorrelation is not only an important tool in spatial data analysis but also a fundamental tool in multidisciplinary research.

DOI: 10.7717/peerj-cs.3520/table-4

This study, which involves human participants, was reviewed and approved retrospectively by the Academic Committee of the Institute of International Rivers and Eco-security. An informed consent waiver was granted by the same committee due to the anonymous nature of the survey and the minimal risk involved.

Automated evaluation results

We evaluated the generated responses via SBERT (a sentence embedding method for measuring semantic similarity) and METEOR (an evaluation metric for text generation that considers recall and multilevel matching). The SBERT similarity and METEOR scores for the generated responses of Model B and Model C (G-Pro Bot) are shown in Table 5. Model C outperforms Model B in terms of both evaluation metrics.

• 1.

  SBERT similarity:

  The G-Pro Bot outperforms Model B on 15 out of 22 questions, whereas Model B performs better on only seven questions. Notably, G-Pro Bot achieved significantly higher similarity scores (defined as a difference ≥ 0.1) in Q6, Q11, and Q15, whereas it scored significantly lower in Q4.

  Focusing on general questions, the G-Pro Bot performed well for eight questions compared with only five questions for Model B.

• 2.

  METEOR scores:

  Compared with Model B, the G-Pro Bot achieved higher METEOR scores for 16 out of 22 questions, whereas Model B outperformed the G-Pro Bot by just six questions. Significant advantages for the G-Pro Bot were observed for Q2, Q5, Q7, Q11, Q15, Q16, Q17, and Q21, whereas Q10 and Q18 had significantly lower scores for the G-Pro Bot. In the subset of general questions, G-Pro Bot performed better for nine questions, whereas Model B was better for only four questions.

Human evaluation results

In our initial qualitative review, evaluators observed distinct performance patterns: Model A produced clear but shallow instructional answers; Model B generated short and often incorrect responses; and G-Pro Bot delivered comprehensive, coherent explanations with strong reasoning, albeit with some verbosity. To complement this preliminary assessment and quantify these observed patterns, a structured error breakdown was conducted based on the error types identified during the initial review. Each model’s response to the 22 questions was manually examined for four distinct error types: Hallucination (generating factually incorrect or non-existent information), Omission (failing to include key information present in the source), Reasoning Error (making logical fallacies), and Irrelevant Detail (including information not pertinent to the question, contributing to verbosity). The results are presented in Table 6 and Table S1.

The findings provide a quantitative confirmation of G-Pro Bot’s superiority. Overall, G-Pro Bot (C-Model) committed the fewest error, compared to the baseline. A breakdown of key error types reveals:

• 1.

  Omission: G-Pro Bot produced only five omission errors (Q11, Q12, Q16, Q18, Q22), compared to 10 for the A-Model and 21 for the B-Model.

• 2.

  Hallucination: G-Pro Bot had two instances of hallucination (Q11, Q22), which was significantly lower than the B-Model’s six instances.

• 3.

  Irrelevant detail: The five instances of irrelevant detail for G-Pro Bot provide a quantitative measure for its previously noted verbosity.

• 4.

  Of its 12 total errors, G-Pro Bot committed only 2 on the 13 general questions, with the remaining 10 occurring on the five specific questions.

G-Pro Bot’s primary advantage lies in its significant reduction of Omission errors. It produced only 5 such errors, a stark improvement over the A-Model (10) and the B-Model (21). Furthermore, while not entirely immune to Hallucination, G-Pro Bot (two instances) demonstrated better factual grounding than the B-Model (six instances). Notably, the B-Model underperformed across all categories.

To complement this objective error analysis, we examine the quantitative data collected through the questionnaire. A total of 62 questionnaires were collected according to the methodology described in the Human Evaluation Methodology section. After the responses were filtered out with a completion time of less than 10 min, those with identical answers for all options and those for which the attention check question failed, a total of 56 valid questionnaires were retained. The results of the questionnaire analysis are presented in Fig. 6.

• 1.

  Comprehensiveness: Model C (G-Pro Bot) was able to provide comprehensive answers for most questions (Q1–Q10, Q12–Q15, and Q17–Q23), outperforming both Model A and Model B. However, on Q11 and Q16, as shown in Fig. 6, compared with Model A, the G-Pro Bot exhibited the same and slightly lower performance.

• 2.

  Diversity: In terms of diversity, the G-Pro Bot outperformed the other two models across all the questions, with the exception of Q16, where it was marginally inferior to Model A.

• 3.

  Empowerment: The G-Pro Bot achieved the highest scores in empowerment across all the questions, with the exception of Q16.

• 4.

  Directness: Model A and Model B performed well in directness across different questions.

• 5.

  Performance on general questions: For the general GIS questions (Q1–Q11, Q18, and Q19), compared with Models A and B, the G-Pro Bot achieved significantly better overall performance.

• 6.

  Expert evaluations The responses from GIS professionals were consistent with the overall findings, as shown in Fig. 7.

Validating the reliability of automated evaluation

This section assesses the reliability of SBERT and METEOR as automated evaluation metrics by examining their alignment with human assessments. To elucidate this relationship, we focus on questions where compared with other models, the G-Pro Bot model received notably higher or lower SBERT or METEOR scores (see Table 7). This targeted comparison elucidates how variations in automated scores indicate perceived response quality.

Our analysis reveals a nuanced relationship between automated metrics and human evaluation. We found that SBERT and METEOR scores strongly correspond with human judgments on content-focused dimensions: comprehensiveness, diversity, and empowerment. For instance, in responses to questions like Q2, Q5, Q6, Q7, Q15, Q17, and Q21, G-Pro Bot’s significantly higher automated scores were mirrored by superior human ratings in these three areas. This indicates that these metrics are effective at identifying semantically rich and informative answers.

However, the evaluation also highlighted the irreplaceable role of human judgment in capturing nuances that automated metrics miss. For example, for questions Q4, Q10, and Q18, G-Pro Bot received lower automated scores, yet was still preferred by human evaluators. This suggests that our model can deliver valuable content in ways not fully captured by similarity-based metrics, reinforcing the need for a complementary human assessment.

The most significant limitation of the automated metrics appeared in the evaluation of stylistic qualities, particularly directness. Across nearly all questions, G-Pro Bot was rated lower on directness by human evaluators, even when its SBERT and METEOR scores were high. This pattern confirms that semantic similarity metrics are not designed to measure conciseness. While a response can be comprehensive and contextually relevant (leading to a high automated score), it may simultaneously be verbose and lack the brevity that users value.

These findings underscore the need for more holistic evaluation frameworks that integrate both automated and human perspectives. Such future frameworks would benefit from validation across a larger, more diverse question set and with evaluators from multiple institutions to strengthen the generalizability of the results.

Strengths of the G-Pro bot

The evaluation results consistently highlight the excellent performance of the G-Pro Bot across multiple dimensions, revealing the significant advantages of the GraphRAG approach.

Enhanced semantic comprehension: The superior performance in automated metrics is not merely a statistical victory; it reflects a fundamental advantage of the GraphRAG architecture. By organizing knowledge into a relational graph, our system moves beyond the simple vector similarity search used in traditional RAG. This allows for the retrieval of a more cohesive and contextually complete set of information before generation, which logically results in a higher semantic alignment (measured by SBERT) and structural integrity (measured by METEOR) when compared to the ground truth answers.

Superior knowledge synthesis and comprehensiveness: The significant reduction in ‘Omission’ errors, as identified in our structured error analysis, offers the most direct evidence for G-Pro Bot’s superior knowledge synthesis and comprehensiveness. This finding is not an isolated observation but rather a direct manifestation of the GraphRAG framework’s architectural advantage. Unlike traditional RAG systems that rely on retrieving isolated text chunks and often fail to synthesize information scattered across a corpus (Chen et al., 2024), GraphRAG constructs a relational knowledge graph. This structure allows the model to traverse interconnected concepts, facilitating a holistic synthesis of information from multiple sources into a single, coherent response. Our work empirically validates what recent studies have theorized (Peng et al., 2024; Zhang et al., 2024b; Han et al., 2025): modeling relationships between entities is the key to a deeper understanding of a knowledge base. Therefore, this significant reduction in information gaps confirms that advancing from simple vector similarity searches to a graph-based retrieval approach is crucial for building effective domain-specific chatbots.

Robustness in handling general questions: A notable strength of the G-Pro Bot, substantiated by our error analysis, is its exceptional performance on general questions. This superiority is concentrated in the 13 general question (Q1–Q11, Q18, Q19). Across this category, G-Pro Bot committed only two minor errors, demonstrating remarkable robustness in synthesizing broad conceptual knowledge. This stands in stark contrast to the baseline models, which frequently failed on these same questions due to numerous ‘Omission’ and ‘Hallucination’ errors (see Table S1). This finding provides strong evidence that the GraphRAG approach, by organizing broad concepts and their relationships into a knowledge graph, enables the model to traverse interconnected information and construct more robust and comprehensive answers for questions that are not tied to a single, specific document chunk (Ngangmeni & Rawat, 2025; Zhang et al., 2024b).

Limitations and areas for improvement

Despite its overall strong performance, the G-Pro Bot presents several areas for improvement that highlight avenues for future research.

Misinterpretation and limitations in retrieving specific operational details: While G-Pro Bot demonstrated strong overall performance, our error analysis identified a specific limitation in retrieving fine-grained operational details. This issue was most apparent in responses to queries about specific software operations (e.g., Q18, Q22), where the model’s answers were often conceptually correct but lacked the precise, actionable steps a user would require. This occasionally led to ‘Omission’ errors where key procedural information was missing. This finding highlights a clear area for future improvement: refining the knowledge retrieval mechanisms to better balance high-level concepts with specific, procedural instructions.

Improving directness and conciseness: Our initial evaluation revealed a tendency for G-Pro Bot to generate verbose, article-style answers. Acknowledging this limitation, we conducted a follow-up experiment to address this issue. We implemented two key changes: (1) prompt engineering, where we instructed the model to provide concise and direct answers by omitting introductory phrases and focusing on key points; and (2) parameter configuring. Specifically, we adjusted key generation parameters by reducing max_tokens to 1,024, adjusting temperature to 0.3, and setting repetition_penalty to 1.2 to discourage redundancy. We then regenerated answers for all 22 evaluation questions using this refined configuration. A qualitative assessment conducted by the authors confirmed that these changes yielded a significant improvement. The new responses are substantially more direct and less verbose, while still retaining the essential information required to answer the questions. An example of this improvement is provided in Fig. 8, and the full set of ‘before’ and ‘after’ responses is available in the Supplemental Material. This successful test confirms that the model’s verbosity can be effectively controlled, and future work will focus on systematically optimizing these settings, perhaps even allowing users to select their preferred response length.

Computational complexity and its trade-offs: The primary trade-off of our accessible, high-quality approach is its computational complexity and consequently slower retrieval speed. Unlike traditional RAG, which retrieves simple text chunks, GraphRAG’s retrieval phase involves navigating complex graph structures. This multistage process is the key factor behind the slower response time of our G-Pro Bot. To address this limitation, we plan to explore the use of more efficient Natural Language Processing (NLP) tools, such as SpaCy, to automatically and quickly construct the knowledge graph. This approach could significantly reduce the computational overhead during the knowledge graph construction process, improving the overall response speed of the G-Pro Bot.

Limited scope of the knowledge base: While the 12 selected documents provide a solid foundation for this proof-of-concept study, they do not encompass the full breadth and depth of the highly interdisciplinary GIS field. Consequently, G-Pro Bot’s expertise is confined to the topics covered in this corpus. Future work should focus on significantly expanding the knowledge base with a wider range of literature—including advanced textbooks, recent research articles, and diverse software documentation—to enhance its comprehensiveness and its ability to address more complex, interdisciplinary queries.

Theoretical and practical implications

The contributions of this research are significant on both theoretical and practical levels, as discussed below: