High statistical noise limits conclusiveness of ranking results as a benchmarking tool for university management

Regression analyses of results from the Times Higher Education (THES)-Ranking and Shanghai University’s Academic Ranking of World Universities (ARWU)-Ranking from 2010-2014 show fluctuations in the rank and score for lower scoring universities (below position 50) which lead to inconsistent “up and downs” in the total results, especially in the THES-Rankings. Furthermore year-to-year results do not correspond in THES- and ARWU-Rankings for universities below rank 50. We conclude that the observed fluctuations in the THES do not correspond to actual university performance and ranking results are thus of limited conclusiveness for the university management of lower scoring universities. We suggest that THE and ARWU alter their ranking procedure insofar as universities below position 50 should be ranked summarized only in groups of 25 or 50. The year to year changes in the ARWU scores are very small, so essential changes from year to year could not be expected, so therefore we argue to publish the ranking less frequently. Additionally, we argue for introducing a standardization process for ranking data in both rankings by using common suitable reference data to create calibration curves represented by non-linearity or linearity.