Citations and the h index of soil researchers and journals in the Web of Science, Scopus, and Google Scholar

Number of papers, citations and h index

Table 1 shows the statistics of h index, number of publications, number of citations, and year of the first paper for 340 soil researchers in the three databases. Our data encompass a wide range of researchers from early-career to well-established and highly-cited researchers. The database is much larger and more diverse than previous studies where a small and focussed group of researchers was used to compare citation metrics between the databases (e.g., Franceschet, 2010; Meho & Rogers, 2008; Patel et al., 2013).

The median number of papers for the 340 soil researchers ranged from 23 (Web of Science) to 79 (Google Scholar) with Scopus having intermediate values. The number of citations is also highest in Google scholar, with a median of 866 citations per author whereas it is 291 in the Web of Science. The h index and its annual increase are lowest in the Web of Science. This pattern holds for all of the metrics presented here: Google Scholar has the highest numbers and the Web of Science the lowest whereas the Scopus numbers are in between. Part of this may be the different types of publications included and also the periods of time covered by the 3 databases are slightly different.

A simple linear regression without intercept was performed between the citation indices of the three databases (Table 2). Google Scholar has on average 2.3 times more articles and 1.9 times more citations than the Web of Science. The Scopus database (all years) has 1.1 times more papers than the Web of Science but a similar number of citations compared to the Web of Science. Since the citations are more correct and complete after 1995, a revision was made to the relationship for post 1995 authors; it shows that Scopus has about 1.2 times more citations than the Web of Science. The 20% higher citations are consistent with the findings by Falagas et al. (2008) in the field of medicine. Similarly, for articles in medical journals, Kulkarni et al. (2009) found that Google Scholar and Scopus retrieved more citations compared to Web of Science (1.22 and 1.20 times respectively).

The relationship between number of papers and citations is scattered, especially for the number of papers (Fig. 1), but the relationships between h index values across the 3 databases appear to be quite linear. The h index in Google Scholar is on average 1.4 times larger than Web of Science, and the h index in Scopus (post 1995 authors) is on average 1.1 times larger than Web of Science. However, for pre-1995 authors, their Scopus h index is similar and sometimes can be smaller when compared to Web of Science. While Google Scholar contains more grey literature (informally published written material) and its citations may contain errors (Harzing & van der Wal, 2009), the h index appears to be quite robust and comparable with Web of Science and Scopus. This is due to the fact that h index does not vary greatly if the number of articles increases (e.g., book chapters or unrefereed articles). In addition, extra citations do not have a large effect on the h index, as once a paper has reached h citations additional citations to that paper do not affect its value (Franceschini, Maisano & Mastrogiacomo, 2013; Courtault & Hayek, 2008).

Our results are different from the study by Franceschet (2010) who evaluated 13 computer scientists from his university’s department and found that on average Google Scholar had five times more papers, eight times more citations and a three-fold larger h index. Our results from 340 soil researchers are more in line with De Groote & Raszewski (2012) who looked at 30 researchers from nursing and found that the h index from Google Scholar is 1.3 times larger than the Web of Science, and Scopus is 1.1 times larger than the h index in the Web of Science. Similar results were obtained by Meho & Rogers (2008) who evaluated 22 human–computer interaction researchers from the UK and found that the h index in Google Scholar is on average 1.6 times higher than Web of Science. Patel et al. (2013) compared publications and citations for 195 Nobel Laureates in Physiology and Medicine using the three databases. They found no concordance between the three databases when considering the number of publications and citations count per Laureate. However, the h index was the most reliably calculated bibliometric index across the three databases.

We calculated the Spearman’s rank correlation (ρ) of the h index of the 340 researchers from the three databases. The three databases show excellent correlation for the h index, with WoS and GS as having the largest concordance. Unexpectedly, the rank correlation in terms of no. citations and no. papers (Table 3) also indicates that the three databases are comparable. This implies that the ranking of individuals within a database is comparable with the other databases. Our correlation is also much higher compared to the 13 computer scientists studied by Franceschet (2010) who only obtained ρ = 0.65. We used a much larger dataset, and the GS data came from the page that was created by the researcher, thus the listed papers and citations are assumed to be more complete.

The h index of soil researchers

In an earlier paper (Minasny, Hartemink & McBratney, 2007) we investigated the relationship between the h index of 228 soil researchers and found that the index was 0.7 times the number of years since the first publication (which we called scientific age, or t). That means if a researcher has been publishing for 10 years his/her h index should be about 7. We calculated this index using the Web of Science database, and now we repeated this using analysis to the 340 soil researchers in this study (which are different from the previous list) (Fig. 2). Although the data are scattered the relationship holds: ${\displaystyle h\text{ index}=0.73\times \text{ scientific age}(R^2=0.72).}$ The Web of Science database shows that the average rate of h index increase over time (m) is 0.7, with the lowest value of 0.06 and highest value of 2.9 (Table 1). The average m value for Scopus is 0.7 and for Google Scholar it is 0.8 (Table 1).

McCarty & Jawitz (2013) evaluated the linear relationship between scientific age and h index for 4 disciplines and found the following mean m values of 0.83, 0.47, 0.43, and 0.36 for biochemistry, water, economics, and anthropology, respectively. Thus the trend of soil science is in between water and biochemistry.

For selected researchers, we tried to calculate the distribution of m (h index divided by the number of years since first publication) as a function of sub-disciplines in soil science. Table 5 shows the distribution of m for WoS and GS according to 6 sub-disciplines. It shows that h index varies between sub-diciplines, for WoS, soil biology, biogeochemistry and ecology have the highest m values (median of 0.8). This is followed by soil physics, soil fertility and management, soil geography and pedometrics, chemistry and lastly pedology (average m = 0.5). The order in Google Scholar is slightly different, but it is consistent in that soil biology has the highest m value and pedology is the lowest. Therefore within soil science, the sub-disciplines also vary in terms of h index. The citation ratios are: Soil biology, ecology and biogeochemistry, Soil management and fertility, Soil geography & pedometrics, Soil physics, Soil chemistry and mineralogy, Pedology 1: 0.9: 0.8: 0.8: 0.8: 0.6; respectively.

Although the number of citations for researchers across the three databases can be quite different, the relationship between the number of citations and the h index is quite consistent across the three databases (Fig. 3): ${\displaystyle h\text{ index}=½n^{½}(R^2=0.95).}$ This relationship follows the function postulated by Hirsch (2005) where the number of citations is about 3 to 5 times h², and it appears the h index follows an absorption-type relationship (Warrick, 2003), increasing rapidly at low numbers of citations with the rate decreasing with increasing number of citations.

Table 4 shows the relationship of the average number of papers and citations per year for the 340 soil researchers. This can be interpreted as: on average, a soil researcher produces 5 articles per year, 2 articles in international refereed journals, 1 in a conference proceedings, and 2 other unrefereed publications. The researcher receives 65 citations per year from journal articles, an additional 13 citations from conference proceedings and another 44 citations from other publications.

Self-citations

A way to boost the h index is by self-citation. Hyland (2003) found that self-citation is 12% of all references in biology, engineering and physics, compared to 4% in sociology, philosophy, linguistics, or marketing. For soil science journals, we found a mean of 12% self-citations but it differs between the sub-disciplines (Minasny, Hartemink & McBratney, 2010). High rates of self-citation were accompanied by high journal impact factor ranking; China and the USA had the highest rates of self-citation whereas Egypt, Algeria, Ukraine, and Indonesia have low levels of self-citations in soil science (Minasny, Hartemink & McBratney, 2010).

So high rates of self-citation may influence the h index and the Scopus database allows calculation of the h index with and without self-citation. Self-citation here is the so-called diachronous kind (Lawani, 1982), which is self-citation from the citations received by the author. The other type is called synchronous which is more difficult to calculate, i.e., author’s self-referencing relative to the total number of references cited in a paper.

The relationship for the 340 soil researchers is consistent and the average h index without self-citation is about 12% lower (Fig. 4): ${\displaystyle h\text{ index without self-citation}=0.88\times h\text{ index }(R^2=0.97).}$

We found a weak relationship between percentage of self-citation and scientific age (t) (Fig. 4). It suggests that some younger authors appear to have high rates of self citation as their works were not known widely and their citations mainly come from themselves, as the researchers mature their papers are more widely known and more external citations were gained thus a lower percentage of self-citations: ${\displaystyle \text{Percent self-citation}=42-5t^{0.5}(R^2=0.18).}$

Journal citations

We retrieved 31 Soil Science journal impact factors (IF) and other metrics from the 2012 Thompson Reuters Journal Citation Reports (JCR, released in June 2013). Google Scholar also has measures of the journal’s metric, the h5-index, which is the h index for articles published in that journal for the last five years. The list of journals for the soil science discipline in Google Scholar is slightly different from the Thompson Reuters Journal Citation Reports (JCR), and therefore we used the journals listed in JCR as the basis for comparison. We searched for the h5-index for the journals in Google Scholar metrics for 2012 (released July 2013).

Table 6 shows that Google Scholar h5-index has a better correlation with the five year IF (impact factor) than the two year IF, and Fig. 5 shows the comparison between GS h5-index and the five year IF. While the h5-index and five year IF have a high rank correlation (ρ = 0.90), the ranking is different for different journals. The journals ‘Soil Biology and Biochemistry’ and ‘Plant and Soil’ both consistently ranked no. 1 and 2 in JCR and GS while other journals appear to be slightly different in their ranking (1 to 3 places difference). The top two journals are able to maintain a large number of citations relative to the number of papers they publish.

There are 4 journals that are ranked much higher ( > = 4 difference in rank) in Google Scholar compared to the IF: ‘Soil Science Society of America Journal’, ‘Journal of Plant Nutrition and Soil Science’, ‘Pedosphere’, and ‘Revista Brasileira de Ciência do Solo’. All these journals are published by national soil science societies (USA, Germany, China and Brazil). In the case of ‘Revista Brasileira de Ciência do Solo’ which ranked 12 in GS and 25 in JCR, Google Scholar includes more citations from non-English articles. Contrarily, there are four journals that are ranked much lower ( < = 4 difference in rank) in Google Scholar: ‘European Journal of Soil Science’, ‘Soil Use and Management’, ‘Journal of Soil and Water Conservation’, and ‘Soil Science’.

The Thompson Reuters Journal Citation Reports suffers from a miscalculation, for example, ‘Australian Journal of Soil Research’ was reported to have a 2 year IF of 3.443. This is a miscalculation, as the journal changed its name to ‘Soil Research’ in 2011, and the IF calculation for Australian Journal of Soil Research only accounts for papers published until 2010. ‘Soil Research’ was again listed as a separate journal in JCR. We have recalculated the actual impact factor for this journal in our analysis.

While there is a positive correlation between cites (citations in 2012 to papers published from the previous 5 years) and IF, we can see that there are 2 trends (Fig. 6A). For journals that published <700 papers between 2007 and 2011 (or on average less than 140 papers per year) IF tends to increase rapidly with increasing citations (1.2 increase in IF per 1000 citations). For the other 7 journals that published more than 700 papers, the slope is half as much (0.6 IF increase per 1000 citations). So there is a drawback for journals that publish more papers. Meanwhile the h5-index is mostly controlled by number of citations following an absorption relationship (Fig. 6B). Although the citations come from WoS, the h5-index still holds the square-root relationship supporting its robustness.

Table 6 also shows that the GS h5-index is more correlated to the Eigenfactor metric compared to IF. The Eigenfactor metric is based on the Google PageRank algorithm which calculated the “influence” of the journal based on the citations weighted by the “quality” of the journal (Bergstrom, 2007). A citation from a highly cited journal will have a higher weighting than a lower cited journal. We showed that this metric is less vulnerable to self-citation than the impact factor (Minasny, Hartemink & McBratney, 2010). Interestingly, Google Scholar does not apply its PageRank for citations.

Vanclay (2007) and Courtault & Hayek (2008) established that the h index is robust and is relatively unaffected by grey literature and errors in citations such as in Google Scholar. Most of the errors (and distortions) in citation databases are found in the ‘long tails’ of the citation distribution and they tend not to affect the h index much. For the journals considered here, we calculated the ratio between h5-index and number of papers, and it shows that only 1–9% (median 5%) of the total papers that contributed to h5-index. In other words, less than 10% of the cited papers are influencing the h5-index. We also demonstrated that the h5-index keeps its relationship even when using WoS citations.

Harzing & van der Wal (2009) recommended the use of the GS h index for Management and International Business journals. We also concur that the h5-index is a better measure of a journal’s citation performance than the impact factor as it is more robust and less affected by citation manipulation. It is now acknowledged that there are ways of manipulating impact factor, which include self-citation, and editorials that listed references to previously published articles (Falagas et al., 2008). The h index is less sensitive to the increase in number of citations, while individual highly cited papers can artificially increase the impact factor. In addition, it only considers the top influential h papers in the journal, thus it does not penalise a journal for publishing a larger number of papers. Although h index can also be manipulated by self-citation, in order to increase the h index considerably, a journal has to be more tactical by increasing a significant number of citations to certain papers.