An empirical study of software ecosystem related tweets by npm maintainers

Studies on software ecosystem

A large number of interdependent packages, for example, increases the risk of incompatibilities within an application. Several recent studies have examined software ecosystem from the perspective of updates (i.e., updating an existing dependency to a more recent version) and migrations (i.e., replacing, removing, or adding a new package dependency) (Cox et al., 2015; Bogart, Kästner & Herbsleb, 2015; Bogart et al., 2016; Decan, Mens & Constantinou, 2018; Abdalkareem et al., 2017; Kula et al., 2018; Decan, Mens & Grosjean, 2019; Cogo, Oliva & Hassan, 2021; Jafari et al., 2023), security (Zimmermann et al., 2019; Kabir et al., 2022; Wyss, De Carli & Davidson, 2022; Alfadel et al., 2023) but the focus has not been on the package manager itself. According to Bogart, Kästner & Herbsleb (2015), developers can make decisions about migration based on awareness mechanisms that incorporate various definitions of stability. Bogart et al. (2016) describes a number of reasons why developers fail to update, including community values, such as policies, support infrastructure, and accepted trade-offs to negotiate dependency changes. A study by Jafari et al. (2023) on how different characteristics of the npm package can influence the predicted update strategy reported that dependent count, age, and release status are the most influencing features. Cogo, Oliva & Hassan (2021) investigated npm dependency downgrades and found that the reasons for reactive downgrades were defects in a specific version of a provider, unexpected features of the provider, and incompatibilities. As reported in another study (Kula et al., 2018), 69% of developers were unaware of the need to update and were not likely to prioritize an update due to its perceived additional labor requirements. Wyss, De Carli & Davidson (2022) identified and characterized 6,292 shrink wrapped clones and found that up to 2,159 depended on outdated and vulnerable npm package dependencies. Furthermore, 207 clones contained vulnerabilities that were not detected by npm audits. Abdalkareem et al. (2017) examined the reasons why developers use trivial packages in the npm ecosystem and found that developers consider these packages to be well tested and well implemented, making them more productive because they do not need to implement small features. Zimmermann et al. (2019) conducted a case study on npm security and found that there are a number of single points of failure with npm and that unmaintained packages pose a threat to large code bases. These studies have demonstrated that developers have difficulties in managing dependent packages.

Studies on communication platforms

Software developers now rely heavily on different communication platforms such as Twitter and question-and-answer sites such as Stack Overflow (Meldrum, Licorish & Savarimuthu, 2017; Chakraborty et al., 2021) or a project-specific forum such as Eclipse forums (Nugroho et al., 2021) and GitHub discussion (Hata et al., 2022) to coordinate communication and learning, share knowledge, and recruiting activities (Meldrum, Licorish & Savarimuthu, 2017; Begel, DeLine & Zimmermann, 2010). Several empirical studies were conducted on Twitter posts to explore software engineering trends (Guzman, Alkadhi & Seyff, 2017; Gonzalez-Marron, Mejia-Guzman & Enciso-Gonzalez, 2017; Bougie et al., 2011; Fang et al., 2020), recommendation tools support (Sharma et al., 2018), community detection (Lim & Datta, 2012) etc. These prior studies show that Twitter posts are useful in understanding software developers’ challenges. In our previous empirical studies, we looked at software ecosystem-related discussions by developers on question-and-answer sites and found that package management issues are the biggest challenge (Islam et al., 2021, 2023, 2022). However, no study has been conducted so far to examine the tweets shared by developers to reveal the challenges associated with using the software ecosystem and how they differ from question-and-answer sites.

In this study, we empirically examine developers’ (i.e., npm maintainers) tweets related to the software ecosystem in order to better understand how social media is used by them to deal with daily issues related to software development.

Research questions

To guide the study, we formulate the following research questions with their motivations.

• Topic of tweet (RQ₁): What are the main topics that npm maintainers communicate on Twitter? Motivation: The npm maintainers are key decision-makers in software project development. Hence it is important to understand how they leverage Twitter as a medium.

• Nature of tweet (RQ₂): What is the formulation nature of tweets communicated by npm maintainers on various topics? Motivation: Having identified the topics of interest, it is important to know the nature of discussions that happen in the context of these topics, where npm maintainers posts are replied to by others (Guzzi et al., 2013; Yasir et al., 2018). Furthermore, Twitter is an informal communication channel that is not limited to discussing problems and solutions, but also allows developers to post questions on information tweets, which differs from Q&A forums, such as Stack Overflow.

• Sentiment of tweet (RQ₃): What is the sentiment of tweets by npm maintainers on Twitter? Motivation: The motivation of RQ₃ is to get deep insight into developers’ sentiments while sharing npm package ecosystem-related information. Stieglitz & Dang-Xuan (2013) reported that tweet sentiment is an important factor in the diffusion of information on Twitter. Tourani, Jiang & Adams (2014) adopted sentiment analysis technique to identify distress or happiness in a software development team. In this RQ, we aim to test the degree of satisfaction/dissatisfaction while sharing npm-ecosystem related tweets.

Data collection

In this section, we describe our methods for collecting npm ecosystem-related tweets by software developers. The data collection process consists of three steps: (i) Building npm maintainers Twitter ID dataset and (ii) extracting npm maintainers Tweets, and (iii) preparing a sample tweet dataset. Details of the data collection procedure are explained below:

• Step-1: Building npm maintainers Twitter ID dataset- For an in-depth examination of npm ecosystem-related issues, we primarily focused on npm maintainers tweets. To identify npm maintainers, we only select those who performed at least one pull request on software packages published on npm ecosystem. To do so, first, we extracted all npm pull requests through the GitHub API and obtained 123,647 npm maintainers GitHub ID. We subsequently extracted user profile information for each GitHub ID. The output of this step is 14,330 Twitter IDs of npm maintainers interlinked with GitHub IDs.

• Step-2: Extracting npm maintainers tweets- To extract npm maintainers Tweets using the output of step-1, we utilize the official Twitter search API. Afterwards, to filter npm ecosystem-related Tweets we limit our data collection process to tweets that only correspond to specific keywords such as npm, pnpm, npm-install, npm-scripts, npmignore, npm-shrinkwrap obtained from our previous work (Islam et al., 2023). While extracting the data, we found many non-English tweets in the dataset. However, we decided to keep them in the dataset for a couple of reasons. First, we aimed to maintain the dataset’s integrity and inclusiveness, reflecting the actual data we collected during the time of the study. Second, preserving these non-English tweets allows for the potential exploration of cross-language sentiments or topics in future research. The output of this step is 39,425 tweets dataset (D1) related to npm ecosystem by npm maintainers.

• Step-3: Preparing sample tweet dataset- After obtaining the npm ecosystem-related tweets (i.e., dataset D1) posted by npm maintainers, we prepare a sample dataset keeping 99% confidence with interval 3 (Aghajani et al., 2019; AlOmar et al., 2022). The output of this step is 1,176 tweets dataset (D2).

Approach

In this section, we describe the approach we have taken to answer each research question. We highlight that our study’s focus is on qualitative coding to capture topics, nature, and the sentiment of tweets. Qualitative coding provides a more detailed and context-rich understanding, especially considering the inherent complexities of the text in our dataset, such as sarcastic tweets. This methodology allowed us to capture subtle variations in communication that automated techniques (e.g. topic modeling) might miss. The detailed approaches of each research question are outlined below:

Topic of tweet (RQ₁): what are the main topics that npm maintainers communicate on Twitter?

Figures 3 and 4 illustrate the results of our qualitative analysis of the tweets posted by the npm maintainers. As a whole, tweets belonging to the package management category, including topics such as usage scenarios, configurations, feature information, bug reports, software comparisons, and dependencies, were most prevalent (i.e., 58%). This finding is consistent with our previous studies on software ecosystem issues (Islam et al., 2021, 2023). The findings indicate that developers and maintainers are using Twitter to filter and curate information on package management activities associated with the npm ecosystem. Following package management issues, tweets belonging to the notification category, including topics such as media sharing, new releases/progress updates, product promotions, and general news, were the most common (28%). In light of this finding, it appears that npm maintainers utilize Twitter to stay informed, maintain relationships with developers, and keep up to date with their domain knowledge. Finally, tweets that relate to the communication category include topics such as community information and community events, which are the most prevalent (4%). Even though community-related tweets are relatively low in number, the large number of tweets communicated every day for the applications in our data sample suggests the number of relevant tweets is significant, and should therefore be considered by software companies when preparing the next version of their applications. Singer et al. (2013) also reported that it is common for developers to use social media to communicate and collaborate with one another. Therefore, npm maintainers should pay attention to update their information in the software ecosystem community.

Nature of tweet (RQ₂): what is the formulation nature of tweets communicated by npm maintainers on various topics?

Figures 5–7 illustrate the results of our qualitative analysis to identify the nature of tweets. We observe that npm maintainers who feel that Twitter benefits them rely on a variety of strategies for posting content related to software ecosystem. In Fig. 5, developers post npm ecosystem-related tweets in form of information (65%) followed by question (13%) and answer (6%). In contrast of tweet categories vs nature as shown in Fig. 6, we find that notification (79.9%) related tweets are mostly shared in the form of information followed by package management (65%), and community-related issues (48.1%). In addition, we find that 18.7% tweets in package management-related issues are posted in the form of questions by developers. In contrast to tweet topic vs nature as shown in Fig. 7, we find that usage scenario (59.7%), software comparison (66%), configuration (71%), build (75.8%), feature information (80.6%) related tweets are dominantly shared in form of information. In addition, question-type tweets are significant in a bug report (25.8%), usage scenario (22.4%), feature request (21.9%), etc. This finding aligns with our previous study (Islam et al., 2022). It appears that npm maintainers use Twitter more for technical issues (such as package management) than for community-centric events and activities. It is important to note that while social media platforms claim to predominantly support community-related activities, we observe a predominance of technical work-oriented discussions in the form of information. In summary, tweets regarding the software ecosystem concerning package management work have gained greater attention in terms of the volume of information provided.

Sentiment of tweet (RQ₃): what is the sentiment of tweets by npm maintainers on Twitter?

Figure 8, and nine illustrate the results of qualitative analysis to identify the sentiment of tweets. Overall, npm maintainers mostly post tweets with a neutral sentiment (51%) followed by positive (30%) and negative sentiment (13%) as shown in Fig. 8. It is possible that a large number of tweets with a neutral sentiment can be attributed to the proliferation of tweets from senior developers. In contrast of tweet categories vs sentiment as shown in Fig. 9, we observe that package management-related tweets are mostly neutral (61%) followed by positive (30.5%) and negative (17.5%) sentiment. This hints that npm maintainers have better experience in package management when using npm ecosystem. In the notification category, most tweets belong to positive sentiment (51.1%) followed by neutral (43.5%) and negative (5.5%). In the community category, most tweets belong to positive (57.7%) sentiment followed by neutral (23.1%) and negative (19.2%) sentiment. The results indicate that npm maintainers extendably use Twitter for community activities in a positive manner to stay up-to-date with the latest software trends and practices, to expand their software knowledge by learning new stuff, and to maintain relationships with fellow software developers.

Implications

Twitter provides developers and package maintainers with a means of staying up to date with software engineering trends and practices, facilitating their learning, and building relationships among themselves. We investigated how the maintainers community use Twitter in order to support their npm development activities and the challenges associated with using software ecosystems. The results of our study demonstrate that Twitter contains useful information for the npm developers and maintainers community. Based on the results of our study, we provide a list of implications based on the analysis results:

• According to our analysis, the majority of tweets by npm maintainer are related to package management issues, such as usage scenario, configuration, feature information, bug, software comparison, and dependency, etc. This result aligns well with our previous works (Islam et al., 2021, 2023) using Stack Overflow, which also reported that developers struggle mainly with package management issues. The findings indicate that developers and maintainers use Twitter to filter and curate information related to package management issues in the npm ecosystem. Therefore, developers and maintainers should consider informal communication channels such as Twitter as well as formal channels in order to gain a more comprehensive understanding of package management issues. The next most popular category of tweets is notifications, such as updates about a new release, followed by community-related information. This finding is consistent with previous studies (Singer, Figueira Filho & Storey, 2014; Tian & Lo, 2014). In detail, a survey by Singer, Figueira Filho & Storey (2014) reported that many developers use Twitter to stay up-to-date with the latest trends, consume knowledge, and network with each other. Another study by Tian & Lo (2014), found that some software developers are very active in posting software-related content. This implies that informal communication channels such as Twitter are adopted by developers and maintainers as an easy way to reach the largest community in the npm ecosystem since the formal channel offers limited features to developers (Yasir et al., 2018). Hence, if npm maintainers plan to inform any information related to the software ecosystem, they should keep this in mind. Moreover, a developer who is just beginning to engage in social media should consider adopting the above strategies in order to increase their chances of staying informed, learning, and developing relationships. For instance, follow the most active npm maintainer tweets in your field, in order to obtain verified information on package management issues. However, choosing whom you follow incurs a cost in terms of attention; therefore, be careful in your selection of whom to follow.

• Developers and package maintainers who consider Twitter an important tool for their development activities use a variety of strategies to post content on. Information-type tweets are the most common format for posting content on Twitter followed by question and answer. This finding aligns well with previous studies (Hughes & Palen, 2009; Tian et al., 2012; Yasir et al., 2018). Our study results show that npm maintainers prefer to share tweets containing information concerning what they learn about a new technology or related practices. This implies that Twitter helps developers and maintainers stay aware of package management tools and practices, learning support, and plays a role in building a collaborative community through sharing useful information related to npm ecosystem. In addition, developing a recommendation system capable of identifying tweets that would be of interest to many developers (e.g., tweets about package management) in order to help them remain current with recent development and gain new knowledge.

• Sentiment analysis revealed that the majority of tweets about the npm ecosystem were neutral, with a smaller proportion expressing positive or negative sentiment. This finding align well with prior study by Guzman, Alkadhi & Seyff (2017). They reported that the overall sentiment polarity of tweets related to software applications is neutral. This suggests that developers generally have a neutral attitude towards the npm ecosystem, with some expressing enthusiasm or dissatisfaction. As a result of these findings, we can infer how typical development circumstances (such as review of package management features, usage scenarios, etc.,) influence the sentiments of members of a software development team (Tourani, Jiang & Adams, 2014; Bano & Zowghi, 2015; Lin et al., 2018). Furthermore, we can identify how sentiments are spread within the npm maintainers community and how measures and methods can be implemented to minimize negative effects and enhance positive aspects that users and developers will perceive positively. For instance, such analyses can be utilized to understand and rationalize developers’ and package maintainers sentiments toward newly released packages and features, helping software developers plan better for future releases.

Recommendations

Besides the results of our study have shown some practical implications in the software engineering area, we also provide a set of recommendations for developers, package maintainers, and researchers, as follows:

• Package maintainers and developers should prioritize the usage scenario of a package to ensure a positive experience for new users.

• The npm ecosystem should continue to monitor and address any negative sentiment expressed by package maintainers to improve user satisfaction.

• Further research could be conducted to identify the specific issues or features that are causing negative sentiment among developers, in order to address them more effectively.

• The findings of this study also recommend that social media posts, such as Twitter posts, should be considered in conjunction with traditional question-and-answering websites, such as Stack Overflow when evaluating technical discussions relating to software ecosystems.

Internal validity

Threats to internal validity relate to experimental bias and error in conducting the analysis. The first threat is the accuracy of the methods used in this study. We perform manual analysis on a random sample since the dataset size is large. To mitigate this challenge, we prepare representative samples for the tweet dataset, with a confidence level of 99% and an interval of three. Thus, we believe that experimental bias and error in conducting the analysis were reduced. Another internal validity is regarding the manual analysis that may produce the potential subjectivity of the study. However, we intentionally chose this approach for several reasons. Manual analysis not only helps us better understand the sentiment nuances, but also allows us to gain insights into the contextual intricacies that contribute to npm maintainers perceptions. In addition, several prior research works (Williams & Mahmoud, 2017; Nugroho et al., 2021) also utilize manual classification approach to understand different community issues due to domain-specific meanings and expressions in text.

External validity

Threats to external validity relate to the generalizability of findings. In our study, we focus only on Twitter which is one of the largest and most popular social media platforms among end-users. In the context of identifying npm maintainers, it is essential to note that the requirement to link a Twitter account to a GitHub profile may result in the inadvertent exclusion of developers who do not have their Twitter accounts linked. Thus, the results may not represent the other npm maintainers who do not link their Twitter accounts to GitHub. Although we conducted a large-scale study of npm-related discussions on Twitter, the findings may not generalize to other question-and-answer sites and social media platforms. However, our study is consistent with previous works that also utilized Twitter data (Bougie et al., 2011; Guzman, Alkadhi & Seyff, 2017; Fang et al., 2020; Gonzalez-Marron, Mejia-Guzman & Enciso-Gonzalez, 2017).

Construct validity

Threats to construct validity are related to potential errors that can occur when extracting data about npm ecosystem-related discussions. The first construct validity relates to the data collection. We used keywords to identify posts related to the npm ecosystem, but some posts may be incorrectly labelled. To reduce this threat, we created the list of keywords from our previous published article (Islam et al., 2023).

In the qualitative analysis of classifying Tweet types, the posts may be mislabelled due to the subjective nature of our coding approach. Despite annotators resolving disagreements through discussion, the labels might still be incorrect. There are also many potential factors that need to be taken into account. To mitigate this threat, we limit the scope of our study to conversational contents of the discussions themselves, to gain insights into how Twitter is utilized. In addition, we took a systematic approach to validate the taxonomy and the comprehension understanding among the first four authors of this article in several rounds. Only when the Kappa score reaches an acceptable range and thus we are able to complete the rest of the sample dataset.