Cognitive expertise in esport experts: a three-level model meta-analysis

Search strategy

The literature search encompassed the timeframe spanning from January 2000 to December 2023. We conducted inquiries across several databases, namely PubMed, Web of Science, Scopus, Science Direct, ProQuest, EBSCO (Academic Search Complete, SocINDEX with Full Text, APA PsycInfo, APA PsycArticles, Psychology and Behavioral Sciences Collection, OpenDissertations, ERIC, APA PsycTests, APA PsycTherapy), utilizing terms merged within the ensuing Boolean expression (cognition OR cognitive OR perception OR perceptual OR attention OR attentional OR working memory OR executive function OR executive control OR inhibition OR task switching OR multitasking OR Multiple-Object Tracking OR processing speed OR reasoning OR planning OR problem solving OR decision making) AND (experts OR professional players OR top players OR elite players OR gamers) AND (video game OR computer game OR esports). In the event that a Boolean search was not permissible, we endeavored to execute alternative search combinations. Other details of our search strategy referenced the meta-analysis about the cognitive abilities in action video game players or athletes (Bediou et al., 2018; Bediou et al., 2023; Kalén et al., 2021).

The search method for gray literature was based on the approach used by Bediou et al. (2018): (a) Queried specialized databases for gray literature such as PsycEXTRA, ScholarOne, opengrey, and base-search. (b) Searches were conducted within the abstracts of annual conferences. (c) Additional searches were performed using Google Scholar and the Dissertations or thesis Abstracts database. (d) Based on the presently available literature, we had identified some experts in esports’ expertise and attempted to establish communication with individuals, compiling a contact list for communication purposes.

Eventually, we directly contacted 20 authors in this field and inquired about any incomplete or unpublished data related to video games cognitive expertise. Subsequently, we received seven responses out of the 20 requests and obtained the requested data in four cases which met our inclusion criteria.

Selection criteria

The inclusion criteria were: (1) studies comparing cross-sectional cognitive differences between esports experts and amateur players. (2) Experts were either registered professional players, part of an organized team, or high-ranked non-professional players (ranking ≤top 1%). (3) Amateurs had never played in tournaments and possessed lower rankings than experts, and novice was excluded. (4) The inclusion criteria of game genre was parallel with Pedraza-Ramirez et al. (2020), including FPS, TPS, RTS, Fight games, Sports games, MOBA, Mobile, or other game genre (e.g., card games). (5) At least one paradigm was employed to assess cognitive abilities, also including simple reaction time or motor control (Bediou et al., 2023). (6) Age ranged from 18 to 35, Gender was not limited.

Data coding

The data coding section was primarily designed to collect and align the metrics of cognitive tests from different studies in order to calculate effect sizes. The study focused on cognitive ability as the dependent variable, various metrics, such as reaction time (RT) and accuracy (ACC). The RT and ACC of Cognitive tasks’ means, standard deviations, and sample sizes of each group were recorded.

This meta-analysis aims at comparing the cognitive differences between experts and amateur players. In studies with three or more subject groups, such as the research conducted by Valls-Serrano et al. (2022), the novice group was excluded, leaving only the expert and amateur groups. Ultimately, all studies included in present analysis featured a single expert group and a corresponding group of amateur players.

In cases where the data was unavailable, pertinent statistics were obtained from the article or extracted using GetData Graph Digitizer 2.26. The extraction and coding of the data were independently carried out by the two authors, and any discrepancies were resolved through discussion until consistent data were obtained.

Moderators

In meta-analysis, studies typically include many moderators, which can be categorical variables (such as classifications of cognitive abilities) or continuous variables (such as age). Setting categorical variables as moderators allows for observing differences in effect sizes between different subgroups, thereby clarifying the main sources of the overall effect size. For continuous moderator variables, moderator analysis is conducted through meta-regression to test whether the coefficient of the variable significantly influences the overall effect size. The following factors were identified as moderators: professional level, cognitive abilities, dependent variable type, game genre, gender, and age.

Data analysis

As the studies varied significantly in design and multiple effect sizes were extracted, we used three-level meta-analytical models with cluster-robust variance estimation to calculate the total effect size (Fernández-Castilla et al., 2021; Pustejovsky & Tipton, 2022). In the three-level models, the basic sampling variance (level 0) is considered, along with estimates of within-study variance (τ²_within-study) and between-study variance (τ²_{between-study}) representing the heterogeneity of multiple effect sizes within a study and across different studies, respectively. In the context of comparing expert and amateur players, τ²_within-study represents the variance of multiple measures of the dependent variable of the same study, such as variance from accuracy and reaction time. By considering both τ²_within-study and τ²_{between-study}, the three-level model provides comprehensive information for understanding overall effect sizes and identifying sources of heterogeneity. All models were fitted using the R package metaphor, and the robust variance was estimated using the clubSandwich package (Viechtbauer, 2010; Pustejovsky & Tipton, 2022).

Considering the relatively small sample size of the included studies, Hedges’ g was employed as the effect size measure. Interpreting the effect sizes, a small effect size was defined as ≤ 0.20, a range of 0.21 to 0.50 denoted small to medium, a range of 0.51 to 0.80 indicated a medium effect size, and a large effect size was considered as > 0.80 (Lachenbruch, 1989).

Sensitivity analysis

Sensitivity analysis was mainly used to assess the stability of the results. In order to rigorously test the stability of the results of this study, this study implemented two distinct ways: the leave-one-out method and the quartile-based analysis. The leave-one-out method was applied by omitting one specific study in each iteration, subsequently reassessing the overall effect size and its significance to gauge any potential shifts in the meta-analytic conclusions (Borenstein, 2009).

In the quartile-based approach, the lower and upper quartiles of effect sizes, referred to as FL and FU, were computed. Points outside the range [FL - 1.5 * (FU - FL), FU + 1.5 * (FU - FL)] were categorized as outliers. The comparison of effect size metrics, with and without these outliers, aimed to confirm the results’ resilience. An insignificant variance would suggest the robustness of the findings.

Risk of bias

Risk of bias was mainly used to assess the quality of studies included in the meta-analysis. Utilizing the Cochrane tool for bias risk evaluation, each study was individually assessed (Higgins et al., 2019). This task was jointly performed by two autonomous reviewers, with any disagreements resolved through discussion until a unified decision was achieved. The assessment encompassed six facets: participant selection, exposure measurement, outcome assessment blinding, completeness of outcome data, reporting selectivity of outcomes, and confounding factors. A mixed-effects meta-regression was employed to examine if these six quality indicators of the studies influenced the overall effect size, thus exploring the possibility of quality-based effect moderation.

Publication bias

Publication bias was primarily assessed to determine whether the meta-analysis may have included an excessive number of studies reporting positive results. For publication bias investigation, two methodologies were adopted: visual funnel plot analysis and p-curve assessment (Simonsohn, Nelson & Simmons, 2014). The former is a commonly used method to detect publication bias, and the latter is used as a complementary method. Asymmetry in funnel plots was an indicator of publication bias. This visual examination was further augmented by Egger’s linear regression test (Egger et al., 1997), which statistically evaluated the asymmetry in funnel plots.

The p-curve analysis, meanwhile, was focused on detecting selective reporting or “p-hacking” by charting the distribution of significant p-values and analyzing the curve’s skewness (Simonsohn, Nelson & Simmons, 2014). A p-curve that demonstrates a rightward skew is indicative of substantial evidential worth in the meta-analysis outcomes. Conversely, a leftward skew or a curve exhibiting symmetry implies a lack of such evidential substance. The determination of this evidential value was systematically conducted through the application of binomial and continuous methodologies for significance testing.

All statistical analyses were conducted using R software, Version 4.2.3 (R Core Team, 2022), and a significance threshold was maintained at 0.05, with 95% confidence intervals duly reported.

Overall effect size

The overall effect size of the meta-analysis was found to be significant (Hedges’ g = 0.373, SE = 0.148, 95% CI [0.055–0.691], p = .012, k = 142, m = 15, N = 1085). This indicates that video game experts exhibit superior cognitive abilities compared to amateurs. Individual study effect size estimates were shown in Fig. 2.

Regarding heterogeneity, there was between-study heterogeneity (τ²_{between-study} = 0.285, 95% CI [0.093–0.822]) and within-study heterogeneity (τ²_within-study = 0.163, 95% CI [0.106–0.245]). These low heterogeneity (both between-study and within-study) was closed to previous study (Kalén et al., 2021). The Q test for heterogeneity was significant (Q = 542.426, df = 141, p < .001). These findings suggest the presence of heterogeneity in this meta-analysis, and the main source of heterogeneity attributed to differences between studies.

Moderator analysis

The results of Moderator analysis present at Table 2.

Sensitivity analysis

The leave-one-out analyses employed in our research affirmed the consistency of our findings. By sequentially omitting each study and recalculating pooled estimates, these consistently fell within the 95% confidence interval of the collective estimate. Notably, the direction and statistical significance of these recalculated results did not deviate (all ps > .05), underscoring the dependability and steadiness of our study’s conclusions.

In our quartile outlier analysis, the comparison of effect sizes, with and without outliers, resulted in an insignificant difference (z = 0.873, p = .383), bolstering the credibility and stability of the outcomes we obtained.

Risk of bias

In order to assess potential study risk of bias, moderating analyses were conducted using all six indicators of study quality included as moderators. None of the moderating effects were found to be significant. Specifically, the moderating effects of confounding variables (F (2,2.32) =1.33, p = .412), blinding of outcome assessments (F (1,5.26) = 0.844, p = .398), incomplete outcome data (F (1,10.1) = 1.15, p = .309) were not significant. No sensitivity analysis was run for the selection of participants, measurement of exposure and incomplete outcome data, as all studies had the same classification. These findings suggest that the results of this meta-analysis were not influenced by the quality of the individual eligible studies. The quality of the included studies is acceptable.

Publication bias

To assess the presence of reporting bias, the symmetry observed in the funnel plot (illustrated in Fig. 3) was examined. The symmetry was corroborated by the results from Egger’s linear regression test (precision = 0.428 ± 1.072, t (3.61) = 0.399, p = .712). Additionally, the application of the trim-and-fill method indicated no requirement for imputed studies to rectify publication bias. The results of the p-curve analysis, depicted in Fig. 4, showed a rightward skew. The statistical power derived from this analysis was calculated to be 87% (90% CI: 78%–93%). Evaluations for rightward skewness, employing both binomial and continuous statistical approaches, validated the evidential substance in our meta-analysis findings. The binomial test showed a significant outcome (p = .008), and similar significance was observed in continuous analyses, both for the full p-curve (z =  − 10.36, p < .0001) and the half p-curve (z =  − 11.73, p < .0001). Collectively, these analyses consistently indicated an absence of publication bias in present study.

Main finding

This meta-analysis evaluates the difference of cognitive abilities in esports experts compared to amateurs. The results of three-level meta-analytical models with cluster-robust variance estimation indicate that the overall effect size is significant and its magnitude is small to medium (Hedges’ g = 0.373), illustrating that experts did exhibit superior cognitive abilities than amateurs. Specifically, the effect size of cognitive differences between professional players and amateurs (Hedges’ g = 0.462) was larger than that between high-ranked players and amateurs (Hedges’ g = 0.027). Among the nine cognitive classifications, the cognitive expertise of esports experts is mainly reflected in spatial cognition and bottom-up attention. The minor but primary origin of heterogeneity arises from between-study discrepancies, although the overall degree of heterogeneity is low. Through sensitivity analyses, assessment of bias risk, and publication bias testing, the current findings have demonstrated robustness, affirming the reliability of the results obtained in this meta-analysis.

Different effects of moderators

The effect size for cognitive disparities between professional players and amateurs is moderately small to medium (Hedges’ g = 0.462), whereas the distinction between high-ranked players and amateur players lacks significance (Hedges’ g = 0.027). This suggests that the overall effect size in this meta-analysis predominantly stems from the former group. However, the moderating effect of professional levels is not significant. This could be attributed to the inclusion of 12 articles on professional players and only three articles on high-ranked players in this meta-analysis. With the limited number of studies on the latter, conclusive inferences cannot be drawn. Presently, a confident assertion can be made that professional players demonstrate noteworthy cognitive advantages over amateur players.

The role of various cognitive abilities differs notably within the realm of esports. We classified cognitive abilities into nine classifications: perception, bottom-up attention, top-down attention, spatial cognition, task-switching/multitasking, inhibition, problem-solving, verbal cognition and motor control. Spatial cognition exhibits the largest effect size (Hedges’ g = 0.822). Among all these nine cognitive abilities, spatial cognition and bottom-up attention present significant effects, while none of the other abilities are statistically significant. Such a result indicates that spatial cognition and attention contribute significantly to the overall effect size of this study. The attention and spatial cognition advantages of experts have a neurobiological basis. Specifically, experts exhibited enhanced intra- and internetwork functional integrations between the salience network (SN) and central executive network (CEN) (Gong et al., 2016). Within spatial cognition, the most frequently used test is visual working memory. Actually, remembering and updating the spatial positions of multiple opponents and skill cooldown times within the game scenario are crucial. Hence, to some extent, spatial working memory might even be considered a gold standard for talent assessment in esports. Selective reaction time falls under the classification of bottom-up attention. In numerous gaming scenarios, players are required to be selective in their actions based on their opponents’ moves. This often involves selective reaction time. Notably, in terms of motor control, experts do not exhibit a significant advantage. This seems contradictory to our general perception of esports players. In fact, referencing the famous Chunking tasks of Chase & Simon (1973), Baba (1993) ingeniously confirmed through a series of experiments that experts’ superior motor control heavily depends on context and the gaming control tool. Once a new control tool is introduced, the experts’ motor control advantage becomes less apparent. We speculate that experts’ superiority in motor control is specialized and might not be pronounced in general cognitive tasks.

How are players’ cognitive expertise formed? It’s important to note that esports players’ cognitive expertise in spatial cognition is inevitably influenced by long-term gaming experience. According to Bediou et al. (2018) and Bediou et al. (2023), gaming training has a substantial impact on spatial cognition (Hedges’ g = 0.26∼0.45). However, it is evident that current gaming training researches is insufficient to fully explain the large effect size observed between experts and amateurs. This suggests that these players may have already possessed certain cognitive advantages before attaining expert status. Interestingly, superior early cognitive abilities might lead to better performance in early gaming stages. For example, in the study by Jakubowska et al. (2021), which focused on CSGO game training over a period of 28 days, novices’ amplitude of the P300 component in the Attentional Blink (AB) task appeared to be predictive of the level of achievement in later games. A higher P300 wave amplitude signifies better attentional abilities (Vogel, Luck & Shapiro, 1998). This electroencephalographic evidence supports the notion that experts initially possess distinct cognitive neural characteristics.

Age emerges as an important moderating variable for cognitive abilities. In this study, the findings reveal a weak positive moderating effect (β = 0.161) of age on the cognitive differences between experts and amateurs. This implies that as players’ age increases, the differences between experts and amateurs become more pronounced. Notably, participants in this study were predominantly aged between 20 and 25, a stage where cognitive plasticity remains relatively high (Bediou et al., 2023). The increase in age implies more gaming training for professional players, thereby intensifying the differences between the two groups. Additionally, we did not find any significant moderating effects of gender. This could be attributed to the fact that current studies on cognitive expertise in esports predominantly involve male participants or have a majority of males with few females. Furthermore, notable differences among various game genres were not observed. These findings are consistent with previous research indicating no distinctions in attention and multi-target tracking abilities between FPS and MOBA amateurs (Bickmann et al., 2021). Perceived differences between players in different esports disciplines are still noteworthy (Phillips, 2023; Phillips & Green, 2023). Considering the early stage of esports research, we recommend researchers to thoroughly report and match the basic information of age, education and training experience between expert and control groups as comprehensively as possible.

Dependent variable type does exert a significant moderating effect. The effect size for accuracy is significant (Hedges’ g = 0.560), while the reaction time is not significant. Previous meta-analyses often select either speed or accuracy metrics of cognitive tests based on their effect size (Bediou et al., 2018). This approach may result in effect size inflation by focusing solely on the most prominent dependent variables and disregarding others. Consequently, we recommend that future studies report diverse metrics, such as the rate of correct rejections and false alarm rate, or consider metrics that account for the trade-off between speed and accuracy. Employing a three-level model and incorporating multiple effect sizes for the same cognitive test can enhance the quality of meta-analytical outcomes (Fernández-Castilla et al., 2021).