Looking for related posts on GitHub discussions

Communication within the GitHub

Open source software development teams use electronic means, such as emails (mailing lists), instant messaging, or forums, to conduct open and public discussions (Guzzi et al., 2013; Vasilescu et al., 2014; Storey et al., 2016). Those channels are considered a rich source of information about OSS development (Guzzi et al., 2013; Storey et al., 2016; Brisson, Noei & Lyons, 2020). Historically, mailing lists were the preferred channel for project communication within GitHub (Rigby & Hassan, 2007; Guzzi et al., 2013; Vasilescu et al., 2014). However, with the rise of other social communication means, researchers observed that this preference was changing (Guzzi et al., 2013; Vasilescu et al., 2014). Issues repositories and PRs threads play relevant roles in GitHub communities’ communication (Brisson, Noei & Lyons, 2020). However, to meet developers’ requests for an effective communication channel at their disposal for open-ended tasks such as brainstorming (Hata et al., 2022), the GitHub engineering team released Discussions. According to Hata et al. (2022) “the forum is one of the first attempts at addressing this request on the GitHub platform.”

GitHub Discussions is a feature of any public or private repository on GitHub (2022a). It facilitates collaborative discussions among maintainers and the community for a project on GitHub (2022a). GitHub company suggests using Discussions to ask and answer questions, share information, make announcements, and lead or participate in project-specific conversations (GitHub, 2022a). Discussions is a collaborative communication forum for maintainers, code and non-code contributors, newcomers, and users to discuss projects’ use, development, and updates in a single place without third-party tools (GitHub, 2022b, 2022a). Additionally, Discussions is the place where daily conversations can take place separated from discussions specifically targeted toward engineering teams (Issues and PRs) (Hata et al., 2022).

Users, maintainers, contributors, and newcomers can join in a conversation by creating, commenting, reacting, or reading a discussion post (GitHub, 2022a). Discussions users can also search for specific topics in discussion posts (GitHub, 2021b). To do so, they must specify keywords in the GitHub search engine. Users can restrict the search results to the discussion title, body text, or comments by applying correct qualifiers.

Authors must specify the discussion title, body text, and category to create new posts (Fig. 1). The category is a mandatory attribute. It helps organize conversations into predefined classes, allowing community members to chat in the right place and find posts with similar characteristics (GitHub, 2021a). Authorized members can define, create, or delete categories in a repository according to the project needs. By default, Discussions provides five types of categories: Announcements, Q&A, Ideas, Show and tell, and General (GitHub, 2021a). Maintainers can create Announcements posts to share project updates and news. Users can create Q&A posts to ask questions, suggest answers, and vote on the most appropriate feedback. Ideas posts can report or share ideas regarding the project improvements. Show and tell posts discuss relevant creations, experiments, or tests. Finally, General posts address any issue relevant to the project (GitHub, 2021a).

Maintainers report a positive acceptance of GitHub Discussions forum in the communities (https://github.blog/2021-08-17-github-discussions-out-of-beta/). They highlight that the forum enables the growth of the communities in the same place they use to collaborate, improve and increase the community members’ engagement, and separate the issues trackers from questions, feature requests, and general chatting. In addition, maintainers point out they can use the discussion threads to access historical data. They can keep track of the questions already asked, their proposed solutions, and the suggestions made. With this threading support, maintainers can individually address demands without losing them in broader discussions.

Related work

Our research contributes to the growing literature on how software developers use communication channels. While Discussions is not the only primary means of communication for GitHub communities, our focus is aiding maintainers to identify related posts (including duplicate and near-duplicate posts) in the Discussions forum. By doing so, we aim to provide maintainers with the knowledge to carry out their tasks more efficiently.

Several studies have approached the deduplication problem in developers’ communication channels, such as bug-tracking and issue-tracking systems (Alipour, Hindle & Stroulia, 2013; Kukkar et al., 2020; Cooper et al., 2021), Q&A forums (Silva, Paixão & de Almeida Maia, 2018, Wang, Zhang & Jiang, 2020; Pei et al., 2021), and collaborative development platforms like GitHub (Yu et al., 2018; Wang et al., 2019; Lima & Soares, 2022;). Researchers use information retrieval (IR) techniques, machine learning (ML) algorithms, deep learning models (DLM), and even a combination of those to deduplicate data in software engineering contexts. Generally, researchers model the duplicate detection task as a ranking problem, a binary classification problem, or a combination of both. Ranking problems combine features to sort the top-k most similar documents (posts, texts, questions) (Liu, 2011) and predict duplicates. Classification problems classify pairs of documents into predefined categories (e.g., duplicates or non-duplicates) (Zhang et al., 2018).

Previous work highlights that detecting duplicate bug reports is crucial for identifying redundant issue reports in bug and issue-tracking systems (Alipour, Hindle & Stroulia, 2013; Cooper et al., 2021; Kukkar et al., 2020). Bug report deduplication is essential to avoid unnecessary work, which can incur significant human effort and time. Researchers claim thousands of duplicate bug reports are reported daily in typical bug and issue-tracking systems. In this sense, Runeson, Alexandersson & Nyholm (2007) investigated NLP techniques to identify duplicated Defect Reports. The authors claim that about 2/3 of the duplicates can be found using the NLP techniques. Alipour, Hindle & Stroulia (2013) investigated how contextual information can improve bug deduplication. The authors proposed a contextual bug-deduplication method evaluated on the Android ecosystem bug repository. The authors applied machine learning algorithms such as C4.5, K-NN (K Nearest Neighbours), Logistic Regression, and Naive Bayes. Their best result, including textual, categorical, and Android architectural context data, achieved an accuracy of 91% and 0.7284 of Kappa measure. Kukkar et al. (2020) proposed a CNN (convolutional neural network) based deep learning strategy for duplicate Bug report detection and classification. The proposed approach achieved an accuracy rate of 85% to 99% and a recall@k rate (the proportion of items successfully retrieved among the top k results) of 79–94%, setting $k=20$. The authors used six publicly available datasets for the approach training and evaluation. Cooper et al. (2021) proposed TANGO, a method to address the challenge of detecting duplicate bug reports in screenshots and screen recordings. TANGO operates computer vision techniques, optical character recognition, and text retrieval to achieve its goal. The results showed that TANGO accurately ranked duplicate videos in the top-2 returned results in 83% of the tasks. The authors claim that TANGO saves almost 65.1% of humans’ time finding duplicates.

Regarding duplicates on developers’ Q&A forums, several studies also aim to detect duplicate questions in such forums (Zhang et al., 2015, 2018; Wang, Zhang & Jiang, 2020; Mohomed Jabbar et al., 2021; Pei et al., 2021). Zhang et al. (2015) proposed DupPredictor, an approach to detect potential duplicate questions in Stack Overflow. The DupPredictor combines the similarity scores of different features (titles, descriptions, latent topics, and tags). The authors evaluated the approach using a pre-labeled Stack Overflow dataset, which achieved a 63.8% recall rate. Ahasanuzzaman et al. (2016) and Zhang et al. (2017) addressed duplicate detection as a supervised classification problem, utilizing ML algorithms such as decision tree, KNN, SVM (support vector machines), logistic regression, random forest, and naive Bayes. The authors used a pre-labeled dataset to train and validate a classifier to detect duplicates in Stack Overflow and evaluated the proposed approach using the recall rate. Mizobuchi & Takayama (2017) created a Word2vec model using sentences extracted from Stack Overflow. The authors contrasted the model efficiency against a pre-labeled dataset they created. The best recall@k rate achieved was 43.13%, setting $k=20$. Zhang et al. (2018) modeled duplicate detection as a “ranking-classification” problem over question pairs. The authors used a data dump of Stack Overflow to train and evaluate the proposed approach, achieving a precision of 75% to 86% and recall of 66% to 86%. Wang, Zhang & Jiang (2020) used deep learning techniques to detect duplicate questions in Stack Overflow. The authors proposed three different approaches based on Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM) to identify duplicates. The authors also used a pre-labeled Stack Overflow dataset to train, evaluate, and test the classifiers and evaluated the approaches using the recall rate. The approach achieved 76–79% of recall@5. Mohomed Jabbar et al. (2021) proposed DeepDup, a deep learning model for duplicate question detection. The authors also use transfer learning techniques to improve the text-pair duplicates classification task, using pre-labeled datasets from the Stack Exchange sub-communities for Ubuntu and English to train and evaluate the approach, achieving an accuracy of almost 75–78%. Pei et al. (2021) proposed an Attention-based Sentence pair Interaction Model (ASIM) to predict the relationship between Stack Overflow questions. The authors used a Stack Overflow dump to train the software engineering specific domain. The proposed model achieved 82.10% precision and 82.28% recall rates. Finally, Gao, Wu & Xu (2022) proposed a method to detect duplicates in Stack Overflow that learns the semantics of a question by combining text features and source code features. The authors used word embedding and convolutional neural networks to extract textual features from questions and structural and semantic features from source code, achieving a recall rate of 68–79%.

Considering GitHub activities, previous work also highlighted the problem of duplicates in issue reports and PRs on GitHub (Li et al., 2018; Wang et al., 2019; Li et al., 2020). Li et al. (2018) analyzed explicit links in both issues reports and PRs. They reported the importance of such links in identifying duplicates. The authors analyzed 70,686 links that represented duplication relationships, from which 59.03% identified duplicate issues and 40.97% identified duplicate PRs. Wang et al. (2019) claim that the uncoordinated and distributed nature of creating issues and PRs leads to extensive data duplication. The authors claim that duplication considerably increases the workload of project reviewers and maintainers. To automatically identify duplicate PRs, the authors hypothesized that two PRs are more likely to be duplicated if they were created around the same time. They experimented on 26 open source repositories from GitHub with over 100,000 pairs of PRs. Their results showed that their assumption improved the baseline performance by 14.36% and 11.93% in terms of F1-score@1 and F1-score@5, respectively. Li et al. (2020) presented empirical evidence on the impact of duplicate PRs on the software development effort. They observed that the “inappropriateness of OSS contributors” work patterns and the drawbacks of their collaboration environment would result in duplicates. In addition, researchers have been proposing different approaches to address duplicates in GitHub (Li et al., 2017; Yu et al., 2018; Ren et al., 2019; Zhang et al., 2020). Li et al. (2017) and Ren et al. (2019) proposed automatic approaches based on traditional Information Retrieval (IR) and NLP techniques to detect duplicates in PRs. The approach proposed by Li et al. (2017) achieved a recall@20 of 55.3–71.0%. The approach proposed by Ren et al. (2019) achieved the best precision rate of 64.3%. Yu et al. (2018) constructed a dataset of historical duplicate PRs extracted from projects on GitHub-the DupPR dataset-by using a semi-automatic approach. The recall@k rate achieved was nearly 70% when the k was set to 20. Zhang et al. (2020) proposed the iLinker, an approach to detect related issues in GitHub. The authors trained iLinker to learn the embedding corpus and models from the project issue text. The approach achieved recall@5 of 38–51% and recall@10 of 45–61%. Finally, Li et al. (2021) present an approach that combines the TF−IDF and deep learning techniques to detect duplicate contributions in the pull-based model at the time of submission. The method produces a list of candidate duplicate contributions most similar to the new contribution, given the combined textual and similarity scores. Evaluation results show that 83.4% of the duplicates can be identified when using the combined textual and similarity scores, 54.8% when using only textual similarity, and 78.2% when using only similarity scores. Table 1 shows an overview of the pertinent literature.

The mentioned approaches detect duplicates in developers’ communication channels. To the best of our knowledge, this is the first attempt to related posts in Discussions. Our main challenge is related to the vast diversity of repositories hosted on GitHub, which provide different software project contexts. All these contexts highlight the relevance of designing a method suitable for replication in other GitHub repositories. Therefore, to detect related posts in Discussions forums, our proposed approach (1) does not rely on a pre-labeled dataset; (2) does not rely on domain-specific models; (3) is based on general-purpose deep learning models (a Sentence-BERT pre-trained model), and (4) aims at improving precision rates. In the “The RD-Detector approach” section, we propose a method to detect related posts in Discussions.

Preprocessing

Applying text similarity algorithms in discussion post contents requires data preprocessing to optimize the algorithm execution (Zhou et al., 2017). In preprocessing, we first extract attributes from each post in the “Attributes’ extraction” step. The RD-Detector uses the attributes’ values to select the posts that should be evaluated. In this research, we use the repository name and the posts’ IDs, categories, authors, dates, and titles as attributes. In the “Data selection” step, the RD-Detector converts the text to lowercase and selects the posts’ title and body text. In addition, the approach filters discussion posts according to the values of their attributes.

The RD-Detector considers the title and the body of the discussion first posts to calculate the semantic text similarity values between pairs of posts. We do not consider comments and replies because they are feedback or design reasoning about the discussion posts’ main topics.

The following two preprocessing steps, “Code snippets and URL removal” and “Punctuation, emoji, numbers, and stopwords removal,” aim to remove noise data from the content of the posts. We remove the code snippets since the machine learning model is a general-purpose model trained to handle NLP problems. Furthermore, the structure and vocabulary of natural language texts differ from machine codes (Sirres et al., 2018). We parsed the HTML document of each post and removed code snippets and URLs embedded in specific HTML tags, e.g., <code> and <a href>.

The “Punctuation, emoji, numbers, and stopwords removal” step removes punctuations, emojis, numbers, and stopwords from posts’ content. To this end, we use the Python Natural Language Tool Kit (NLTK) library (https://www.nltk.org/) (Bird, 2006). We do not remove the punctuation symbols ‘.’ and ‘_’ because they concatenate words, e.g., ‘Next.js’ and ‘version_1.3’. Finally, we apply lemmatization to the remaining text (Zhang et al., 2017) and eliminate zero-length discussion posts in the “Lemmatization, empty posts removal” step.

Relatedness checker

The “Relatedness Checker” phase is the core of the RD-Detector (Fig. 2B). In this phase, the approach computes the similarity score of post pairs and detects the candidates for related posts (R). The preprocessed data is the input of this phase (Algorithm 1, lines 0–3). R is the output of the “Relatedness Checker” phase (Algorithm 1, line 30). This phase comprises two steps: (1) the similarity measurement (“Similarity checker”) and (2) the “Selection of related post candidates.” The steps are as follows.

Data collection—GitHub Discussions

We collected all public posts of three OSS communities. We stored the collected data in a local dataset. We downloaded the discussion threads’ HTML files (discussion first post and comments) and compiled a list with the posts’ IDs, categories, authors, dates, and titles (the posts’ attributes).

We collected data from Gatsby ( $D_{p=Gatsby}$), Homebrew ( $D_{p=Homebrew}$), and Next.js ( $D_{p=Next.js}$) repositories (Table 2). We chose these three repositories based on three main criteria: (1) the use of GitHub Discussions, (2) the GitHub Discussions engineering team’s involvement collaboration in contacting maintainers, and (3) the availability of OSS maintainers for evaluation. We collected all public discussion posts created in the repositories’ forums until September 2022. In total, we collected 16,048 posts.

Dataset characterization

According to the Gatsby community documentation (Gatsby Community, 2022), “Gatsby is a free and open source framework based on React that helps developers build blazing fast websites and apps.” We call $D_{Gatsby}$ the set of posts collected from Gatsby’s repository. On the date we performed the data collection, we collected 1,886 posts from Gatsby, $|D_{p=Gatsby}|=1,886$ (Table 2). The Gatsby forum contains repository-level posts, meaning that posts are not visible at the organization level. Figure 3A shows the frequency distribution of posts created in the Gatsby community over 32 months (01-2020 to 08-2022). According to this time window, 01-2020 to 08-2022, the average growth rate of forum usage by the Gatsby community was 13.4% (considering the frequency of new posts). Gatsby makes available the following category types community, help, ideas-feature-requests, RFC, and umbrella-discussions. Help posts are most common, totaling 76.03%, followed by ideas-feature-requests, umbrella-discussions, RFC, and community, totaling 19.24%, 1.85%, 1.48%, and 1.37% of posts, respectively (Fig. 3B).

Homebrew is an OSS project that makes it easy to “install the UNIX tools Apple didn’t include with macOS. It can also install software not packaged for your Linux distribution to your home directory without requiring sudo” (Homebrew Project, 2022). We call $D_{Homebrew}$ the set of posts collected from Homebrew’s repository. We collected 2,490 posts from Homebrew, $|D_{p=Homebrew}|=2,490$. Figure 3C shows the frequency distribution of posts created in Homebrew Discussions forum over 24 months (09-2020 to 08-2022). The oldest post collected from Homebrew dates from September 2020. According to data in $D_{Homebrew}$, the average growth rate of the forum usage in Homebrew between 09-2020 and 08-2022 was 18.02% (considering the frequency of new posts). Homebrew organizes its discussion threads at the organization level according to the problem types and not by question types. Homebrew provides the following categories: casks, getting-started, tap-maintenance-and-brew-development, everyday-usage, linux, polls, and writing-formulae-casks. The everyday-usage posts are the most common in the $D_{Homebrew}$ dataset, totaling 47.30%. Followed by getting-started, casks, tap-maintenance-and-brew-development, linux, writing-formulae-casks, and polls totaling 17.46%, 13.65%, 9.27%, 6.90%, 5.34%, and 0.04% of posts, respectively (Fig. 3D).

Next.js is an OSS project that “provides a solution to build a complete web application with React” (Vercel, 2022). The Next.js community has stood apart in supporting the launch of Discussions since the start (https://github.blog/2022-01-13-how-five-open-source-communities-are-using-github-discussions/). Compared to Gatsby and Homebrew, Next.js is the project with the highest number of posts analyzed, characterizing an active OSS community. The Next.js forum contains repository-level posts. Figure 3E shows the frequency distribution of new posts over 32 months (01-2020 to 8-2022). In this period, the average growth rate of the forum in the Next.js community was 30.43% (considering the number of new posts created). The Next.js project has the highest average growth rate among the three analyzed projects. When we collected the dataset, the discussion categories were ideas, help, polls, react-server-components, RFC, and show-and-tell. Help posts are the most frequent in the Next.js dataset, totaling 85.82% of the posts. Followed by ideas, show-and-tell, RFC, react-server-components, and polls, totaling 12.73%, 1.03%, 0.21%, 0.17%, and 0.01% of posts, respectively (Fig. 3F).

Finally, all posts collected compose the dataset D used in this work. Therefore, $D=D_{p=Gatsby}\cup D_{p=Homebrew}\cup D_{p=Next.js}$.

Preprocessing phase applied to discussions dataset

We preprocessed the dataset D. First, we split the dataset into subsets using predefined filters. We considered two filter types: project and category filters. We used the project filter, $p$, to select posts from a specific repository by defining the project’s name. We used the category filter, $c$, to select posts that matched Help and Ideas category types. By setting both filters, we can better evaluate and report the RD-Detector results. Since Help posts tend to have more code snippets than others, it is relevant to consider the posts’ category.

We set both filter values using the discussion post attributes extracted from the input dataset. The category filter values depend on the types of categories provided by the project. We express the use of both filters by $p=\alpha |c=\beta$, where $p$ identifies the project name (ex.: Gatsby, Homebrew, and Next.js) and $c$ identifies the discussion category type.

We standardized the category labels to report our results, making them easier to describe and follow. We use the following labels: Q&A, Ideas, and ALL. The Q&A label refers to ‘Question and Answers’ (help) discussion types. The Ideas label refers to ideas or ideas-feature-requests categories. The Q&A label refers to the help category for both Gatsby and Next.js projects, and the Ideas label refers to the ideas-feature-requests and ideas categories for Gatsby and Next.js projects, respectively. In addition, we created the label “ALL” to refer to all posts from a particular repository, regardless of their original category. For example, the category label ALL comprises all posts pre-classified in ideas, help, polls, react-server-components, RFC, and show-and-tell categories from the Next.js repository. As described in the “Dataset characterization” section, the Homebrew project discussion organization differs from the other projects. Due to this fact, we chose not to filter Homebrew posts according to the category type. Therefore, we report the results of the Homebrew project by setting $c=ALL$.

We considered splitting the dataset into categories because Q&A and Ideas type posts are the majority in both Gatsby and Next.js projects. The union of the Q&A and Ideas types discussion represents 95.28% and 98.55% of the posts in the $D_{p=Gatsby}$ and $D_{p=Next.js}$ datasets, respectively. In addition, Discussions documentation encourages using both categories for different purposes: Q&A-type posts for asking the community for help on project-specific issues and Ideas for sharing and discussing ideas for new features (GitHub, 2022a).

We derived seven sub-datasets by applying different filter configurations to the datasets (Table 3, column 3). We use $D\mathrm{\_}p=Gatsby|c=Q\&A$ to designate the subset of Q&A posts collected from the Gatsby project, $D_{p=Gatsby|c=Ideas}$ to denote the subset of ideas-feature-requests posts collected from the Gatsby project, and so on. We assessed the proposed approach (RD-Detector) over the seven sub-datasets.

Next, the sub-datasets went through cleaning, denoising, and formatting steps. The last preprocessing step ignores zero-length posts. In total, the approach discarded 3, 2, and 6 posts from Gatsby, Homebrew, and Next.js projects, respectively.

Relatedness checker applied to GitHub Discussions

After the preprocessing phase, the RD-Detector calculated the semantic similarity values for each pair of posts (as presented in the “Similarity checker” section), using the seven sub-datasets presented in Table 3, column 3.

The RD-Detector uses the K value to select the similarity values of the top-K most similar posts. In order to evaluate the RD-Detector precision, we set the K value to 5 and 10. For each $D_{p=\alpha |c=\beta }$ sub-dataset presented in Table 3, we configured the approach to run over $K=5$ and $K=10$. This way, we evaluate the approach’s precision by considering the five and the ten most similar posts to every discussion in the dataset. Values greater than 10 expand the search bounds for related posts. We did not consider K values greater than ten because the proposed approach aims to increase precision. However, we conducted a second experiment to measure the recall rate of RD-Detector. To do so, we first manually created a labeled sampling of (un)related posts. Then, we evaluated the RD-Detector considering different values of K. We expect maintainers to set the value of K according to their needs. They can decrease the K value to increase the chances of detecting duplicates or increase K to examine occurrences of near-duplicates.

In total, we assessed the approach over 14 different configuration groups (Table 3, column 4). For each configuration group, RD-Detector computed the local threshold value, $T_{related}$, and detected the set of candidates of related posts, R. $R\mathrm{\_}p=Gatsby|c=Q\&A|K=10$ denotes the set of candidates for related posts detected considering the configurations $p=Gatsby|c=QA|K=10$. One can use the same reasoning for the other 13 sets of related posts in Table 3, column 4. We used the configuration groups to evaluate the RD-Detector and report our results.

The RD-Detector evaluation

We recruited maintainers of the three OSS projects analyzed (one maintainer per project) and three Software Engineering (SE) researchers to evaluate the RD-Detector outcomes. OSS maintainers and SE researchers manually classified pairs containing candidates of related posts as duplicate, near-duplicate, or unrelated. OSS maintainers judged pairs according to their work project. The SE researchers judged pairs containing candidates of related posts detected for the three projects. The maintainers, M_Gatsby, M_Homebrew, and M_Next.js, were contacted via the GitHub engineering team. M_Gatsby actively contributes to Gatsby’s community; he creates content and themes for Gatsby. He has committed to the repository more than 600 times over the past five years. M_Homebrew is Homebrew project leader. He has actively committed to the repository more than 7,000 times over the past 11 years. Finally, M_Next.js is also an active contributor to Next.js’s community; he has committed to the repository more than 500 times. The SE researchers, SE_R1, SE_R2, and SE_R3, have different expertise in software development. SE_R1 is an industry practitioner and a Software Engineering researcher, SE_R2 is an active OSS contributor and a Software Engineering researcher, and SE_R3 is a Software Engineering researcher. All researchers have more than five years of experience in their respective working areas.

Evaluators received online documents containing instructions to evaluate the candidates of related posts detected by the RD-Detector configuring $K=5$ and $K=10$. The documents (1) described the concept of duplicated and near-duplicated posts and (2) listed pairs of related post candidates. We characterized each pair of related post by describing the ID and the title of the master and target posts. The ID was a link from which evaluators could access the original posts. We instructed the evaluators to add the label ‘D’ to duplicates, ‘R’ to neaR-duplicated ones, and ‘N’ for uNrelated posts. We also asked the evaluators to add comments to justify their judgments. Although we consider related posts those posts that are duplicated and near-duplicated, we made sure that the evaluators were well-informed about both concepts, as we presented in Section “The RD-Detector approach”.

We measured the RD-Detector precision rate compared to the OSS maintainers’ and SE researchers’ judgment. Maintainers are aware of their entire project. They could judge those pairs of related post candidates that require in-depth project context knowledge. The SE researchers evaluated pairs of well-known related posts of all projects. The researchers also supported evaluating related post candidates for the Next.js project.

The impacts of changing the K value

By configuring the RD-Detector to run over $K=5$ and $K=10$, we note that the sets containing candidates of related posts created when we set $K=5$ are subsets of $K=10$. Therefore, there is a risk of false-positive predictions propagation. We identified this propagation problem by analyzing the sets containing candidates of related posts created using the configuration groups $p=Gatsby$ and $c=Ideas$. The same unrelated post pair $(master,target)$ occurs in $R_{p=Gatsby|c=Ideas|K=5}$ and $R_{p=Gatsby|c=Ideas|K=10}$. Although the precision rate tends to decrease, the approach detects new pairs of related posts when we vary the K values from 5 to 10.

Lower K values restrict the search space and increase the local threshold value, resulting in higher precision but compromising the detection of all related post pairs. Low K values are beneficial for identifying duplicates, providing evidence for maintainers to remove posts from the community forum. In this scenario, high recall may not be necessary, as maintainers typically seek only related posts that might be duplicates. Conversely, higher K values expand the search space for related posts and reduce the local threshold value, detecting new pairs of related posts and compromising the approach precision. Maintainers can set high K values to evaluate the posts comprehensively. In this case, maintainers might use the RD-Detector results to iterate through the posts (Buckland & Gey, 1994) to detect topics commonly discussed and plan proper interventions. Maintainers interested in conducting a deep analysis of the posts can vary the K value according to their needs. A deep discussion regarding the impacts of changing the K value can be found in the reproduction package (Lima, 2023).

False-positive RD-Detector predictions

Four authors of this article manually analyzed the false positives. We identified some limitations of the proposed approach. We describe the reasons for the false positive predictions based on evidence extracted from the posts.

We analyzed the false-positive predictions and noted that the approach identified posts with similar topics. However, it failed to capture the project issue specificity. We observed that the RD-Detector could fail to treat particular contexts of software projects, suggesting that related posts may address the same project feature but differ on the issue’s specificity. We will call this limitation the “project-specific limitation.”

We also noticed that the posts’ creators used screenshots to detail or describe the issues and used error log descriptions to show the stack trace of where the error occurred. In addition, we observed the predominance of templates’ and projects’ keywords in the false-positive predictions. The RD-Detector removes screenshots and error descriptions embedded in HTML tags during preprocessing. The approach does not use images or error log descriptions as a source of evidence. However, removing both the screenshots and log errors may eliminate the problem specificity. In addition, after preprocessing, the project’s keywords may stand out against the actual post content. We identified the predominance of the template keywords in the false-positive pairs.

Based on this analysis, we can propose improvements to the proposed approach. For example, we can use the maintainers’ judgments to optimize the classifier by providing related and non-related post samples. Furthermore, we can design strategies to minimize project-specific limitations by treating the predominance of project and template keywords and considering screenshots and log error descriptions as sources of evidence.

RD-Detector practical applications

We asked maintainers to comment on their decision regarding the candidates for related posts (duplicates and near-duplicates). Based on their comments, we could identify practical applications of RD-Detector.

• 1. to combine the posts’ content merging the related posts—“These (post) could have been combined into one discussion and it would have made sense…” (M_Homebrew), “…if it were up to me, they should have gone together in the same discussion.” (M_Homebrew).

• 2. to move a post content to another location reorganizing the discussion threads as comments to each other—“…the new issue should probably have been posted as a comment in the master discussion” (M_Homebrew), “…(the posts) would’ve received better traction as a comment on one another.” (M_Next.js), “This discussion could’ve sufficed as a comment on 21633.” (M_Next.js).

• 3. to recruit collaborators for specific tasks—“…could be useful though for people looking for other guides to contribute to (in this instance).” (M_Gatsby).

Although maintainers reported some practical applications for RD-Detector, the approach does not aim to merge or organize posts automatically. However, we envision that the RD-Detector can support maintainers in the decision-making process of merging or organizing the posts by detecting related ones.

The variety of projects on the GitHub platform provides opportunities to develop innovative approaches to detect candidates of related posts in Discussions. However, the significant number of communities and the projects’ singularity challenge developing and validating such systems. Those reasons endorse the RD-Detector design decision to use a general-purpose machine learning model and calculate local threshold values.

Contrasting the RD-Detector measurements

In this section, we compare the relative accuracy, precision, and recall measures obtained by RD-Detector with the methods presented in the “Related work” section. We compare the measurement rates reported in the cited articles with the achieved rates by RD-Detector. We do not focus on identifying which approach is better. We intend to provide evidence of how RD-Detector fits to detect duplicate and near-duplicate posts using a Sentence-BERT pre-trained general-purpose model.

Note that some research evaluated the proposed approaches using precision@k and recall@k measures. Both metrics are typically used to measure the relevance of the top $k$ results to the user’s preferences or needs, especially in scenarios where the system is expected to provide a ranked list of items or results (ranking problems). It should be pointed out that the value of K in this study has a different interpretation than previously explained $k$ (precision@k and recall@k). We use the K value to delimit the search bounds for related posts candidates and create the S distribution. The RD-Detector uses the K value to select the similarity values of the top-K most similar posts to each discussion post in the input dataset. The greater the value of K, the greater the number of similarity values selected. RD-Detector was not evaluated based on the K top recommendations. We aim to assess the entire set of related candidates returned by the approach, as we do not consider R to be a ranked list of related posts.

First, we contrasted RD-Detector accuracy with the accuracy values reported by research that aims to identify duplicates in issue and bug-tracking systems. The RD-Detector accuracy varies between 88% (Gatsby) to almost 92% (Homebrew and Next.js). Alipour, Hindle & Stroulia (2013), Kukkar et al. (2020), and Cooper et al. (2021) reported that their approach achieved 88%, 85–99%, and 84% of accuracy, respectively. The range of values achieved suggests the usefulness of the RD-Detector in identifying duplicates. In addition, Cooper et al. (2021) claim that humans can save 65% of the time spent manually detecting duplicates using their approach. Although we do not have this measure regarding the use of RD-Detector, we believe our approach can also reduce the time spent by maintainers in identifying related posts.

Second, we contrasted the precision and recall relative values achieved by RD-Detector with approaches that identify duplicates in developers’ Q&A forums. Zhang et al. (2015)’s proposed approach, DupPredictor, achieved a 63.8% recall rate. Mizobuchi & Takayama (2017)’s approach achieved recall values ranging from 15.48 to 43.13%. Zhang et al. (2018)’s approach achieved recall between 66 and 86%. Wang, Zhang & Jiang (2020) achieved the best values for recall@20, ranging from 76 to 79%. Finally, Pei et al. (2021) reported achieving a recall of 82.28%. We can notice that the strategies vary regarding the reported recall. According to Table 8, the RD-Detector achieved a recall rate of almost 67% for the Homebrew project. We believe this value is because the RD-Detector uses a general-purpose model. All research contrasted uses pre-labeled bases to train or optimize machine/deep learning models. Creating a labeled dataset for Discussions is a research opportunity. However, the dynamism and variety of software contexts on GitHub are challenging to create such a dataset. The labeled dataset we created to measure the RD-Detector recall rate is available in the reproduction package (Lima, 2023).

Finally, we contrasted the relative recall values of approaches that aim to deduplicate issues and pull requests within the GitHub platform with the RD-Detector achieved recall rates. We observe that the RD-Detector can support maintainers in detecting duplicate or near-duplicate posts in Discussions. Yu et al. (2018) reported achieving 70% recall, Li et al. (2017)’s approach achieved between 55.3 and 71%, and Zhang et al. (2020) reported achieving between 38–51% recall@5 and 45–65% recall@10.

Although the recall rate is a relevant metric, we focus on developing a more conservative approach that detects true positive related posts to support the decision-making process of maintainers. Besides, the RD-Detector does not make automatic interventions such as deleting and merging posts. In addition, a conservative approach prevents maintainers from spending a long time moderating truly related posts. Given that RD-Detector is a parameterizable approach, maintainers can adjust the value of K (increase or decrease) to expand or compress the search space for related posts and, consequently, change the threshold value used to detect related post pairs. In this case, maintainers can configure the RD-Detector according to their respective time availability and interest to analyze pairs of related post candidates, choosing between a more conservative (better precision) or a more exploratory approach.

Based on the precision values reported in Table 7 and the achieved recall rates in Table 8, we claim that RD-Detector is suitable for detecting related posts (duplicate and near-duplicate posts) in Discussions. However, we acknowledge that the recall reported in Table 8 is low for $K=5$ and $K=10$. Our objective is to enhance the approach’s precision and recall values. To achieve this, we intend to upgrade RD-Detector by incorporating code snippets and screenshots that users share in posts during the similarity measurement of posts.

The implications of this research

This research enables opportunities for maintainers and software engineering researchers as follows.

Maintainers: previous research shows that maintainers need to manage multiple aspects of the projects to ensure that the projects’ vision endures (Guizani et al., 2022) and the projects’ long-term sustainability (Dias et al., 2021). To do so, maintainers perform different activities encompassing code and non-coding tasks (Dias et al., 2021; Trinkenreich et al., 2021). Such responsibilities intensify the workload of maintainers. Tan & Zhou (2020) highlight proposing tools as a best practice to decentralize maintainers’ responsibilities. In addition, Dias et al. (2021) report that “to sustain a long-term vision of the project, maintainers should delegate tasks.” In this way, the RD-Detector emerges as an alternative tool to alleviate the maintainers’ labor-intensive task of detecting related posts.

Research opportunities: to address the diversity and uniqueness of communities hosted on GitHub, the RD-Detector is based on a Sentence-BERT general-purpose machine learning model. Besides, the approach uses descriptive statistics to calculate the local threshold to detect related posts. Given that software projects are unique development ecosystems, we can not use a single and universal threshold to detect related posts. Research opportunities arise to understand how communities use Discussions and how post categories differ.