Methods of performance analysis in women’s Australian football: a scoping review

Registration

Consistent with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines (Tricco et al., 2018), this systematic scoping review was registered with Open Science Framework on the 27th of March 2021 and amended on the 30th of January 2023. (https://osf.io/ufczd/ and https://osf.io/vxb8q)

Data sources

A search strategy was created and adapted to the following five databases: MEDLINE, ProQuest Central, Scopus, SportDiscus, and Web of Science. The search was conducted from the earliest record of each database up to the 27th of June 2022. To ensure the thoroughness of the systematic search, grey literature was also searched for through the Proquest Dissertations database. The reference lists of eligible studies were also manually screened for additional relevant studies and reports.

Search strategy

The search strategy was created using the Population Concept Context (PCC) framework to define the research question under investigation (JBI, 2015). The population (P) of interest was women and girls AF players, of any age or level of play. The concepts (C) of interest were performance analysis measures and methods related to the sport’s physical, technical, and tactical requirements. The context (C) included methods that explicitly analysed and reported any match-play performance characteristics. Due to the emergent nature of the research field, a broad set of search terms were selected relating to the PCC framework elements (see Supplementary Information 2).

Inclusion/exclusion criteria

Studies were screened for eligibility by their title and abstract, and then full-text, using a set of inclusion and exclusion criteria created according to the PCC framework elements (see Table 1).

Data management

Data extraction

One reviewer extracted individual study data, which a second reviewer reviewed to verify that no translation errors were made in the extraction process. The extraction of data was collated into a spreadsheet using Microsoft Excel. Data collected included metadata about the paper (authors, year, journal published), study type, population, contextual information of the data and methods used, study objective(s), and performance analysis methods, definitions, and outcome measures used.

Critical appraisal

To assess methodological quality, two independent reviewers performed a critical appraisal of all included studies. The critical appraisal of the research methodology of included studies is essential to the review, despite it not typically being conducted in scoping reviews (Munn et al., 2018). It ensures that studies can be reliably used for reference while establishing a precedent for the appropriate application of methodology for practitioners. As the eligibility criteria for this scoping review included all study designs, the 16 questions presented by Hindle et al. (2020) were applied as it has been previously in other sport science systematic reviews (Natera, Cardinale & Keogh, 2020; Nicol et al., 2022; West et al., 2022). These questions were developed to address the possibility of different designs and assess both methodological quality and risk of bias (Hindle et al., 2020). Each question scored a 1 or 0 dependent on a yes or no answer, respectively, providing a total critical appraisal score of 0–16. Total scores were then calculated into a percentage and graded using the suggested thresholds of quality: low risk of bias (>66%), satisfactory risk of bias (34–66%), and high risk of bias (<33%) (Hindle et al., 2020). Disagreements on critical appraisal question scores were resolved by a third reviewer. The critical appraisal results for each eligible study were then considered when commenting on research questions, method, statistical appropriateness, sample selection, and final conclusions.

Data synthesis

Studies were classified into themes by whether they addressed physical, technical, or tactical measures. Those deemed to address physical performance were further sub-categorised using a selection of physical performance themes previously identified by Johnston et al. (2018) that are specific to AF match-play. Categorisations were modified where themes were captured simultaneously in a performance analysis measurement. Technical and tactical studies were categorised into the performance analysis areas presented by Lord et al. (2020). Explanations of the thematic categories utilised for the narrative synthesis are provided in Table 2.

Performance analysis methods utilised in included studies were deemed to use either descriptive statistics, inferential statistics, predictive modelling, or other modelling methods (including machine learning) to determine relationships of variables to match-play performance measures (Lord et al., 2020).

A narrative synthesis was conducted to summarise findings utilising the categorisations outlined in Table 2 in conjunction with individual study design, data characteristics, contextual variables, and critical appraisal results.

Study selection

Figure 1 presents the stages of the search and screening processes. Following duplicate record removal, the five database and manual citation searches produced 183 records. The screening process resulted in 12 studies included for analysis. The most prominent criteria for exclusion were studies with a lack of match-play measures and research objectives not analysing technical, tactical, or physical performance.

Study characteristics

The 12 identified studies were classified to address themes of physical (n = 9, 75%), technical (n = 5, 42%) and tactical performance analysis (n = 2, 17%). Both tactical studies (Black et al., 2019a; Cust et al., 2019) also addressed technical performance, while two studies addressed both physical and technical measures (Clarke et al., 2018; Clarke, Whitaker & Sullivan, 2021). Investigation of the elite level of play was more prominent (n = 7) relative to the sub-elite level (n = 4), with one study presenting a comparison of these levels of play (Clarke et al., 2019). Only three studies (Clarke, Whitaker & Sullivan, 2021; Cust et al., 2019; Dwyer, Di Domenico & Young, 2022) used data from multiple seasons, with the remaining studies only using a single season sample (n = 9). All studies solely investigating a sub-elite level-of-play (n = 4) were conducted on 2016 season data, while elite level-of-play was analysed in the 2017 (n = 6), 2018 (n = 3), 2019 (n = 2), and 2020 (n = 2) seasons.

Methods of match-play physical performance

Table 3 presents the studies that investigated physical performance measures in women’s AF. All physical performance studies (n = 9) were of an observational nature. Traditional inferential statistics and effect sizes were employed to evaluate relationships between physical performance variables and match-play outcomes or comparison of different population groups in all studies (n = 9). For the purposes of this review, the theme sub-categories of distance, high-speed distance, and sprinting were condensed into one all-encompassing sub-category of running demands, given the high overlap of studies covering elements of each of these measures. Therefore, the physical performance analysis sub-categories utilised were running demands, pacing and peak periods of performance, training measures, player physical qualities (when analysed concerning match-play data in the form of aerobic fitness, speed and acceleration, agility, strength and power, or body composition and anthropometry), the effect of match outcome, player quality and experience, acceleration, match importance (e.g., finals games), and demands associated with physical contact (e.g., tackling) (Johnston et al., 2018). Included studies considered the running demands of the sport (n = 9), pacing and peak periods (n = 5), training measures that are related to on-field performance (n = 2), player quality and experience (n = 2), and player physical qualities measured outside match-play and tested for a relationship to match-play performance (n = 3) including tests of aerobic fitness (n = 3), tests of speed and acceleration (n = 1), and tests of strength and power (n = 1). Match outcome (n = 1) and match-play acceleration (n = 1) were also addressed. No study addressed match importance or contact demands. Subsets of physical qualities of agility and anthropometry were also unaddressed. Two differing thresholds of running speed were utilised across the nine studies examining physical performance (Table 3).

Methods of match-play technical and tactical performance

Table 4 presents included studies that measured technical and tactical performance analysis qualities in women’s AF. The two studies that conducted tactical analysis used a predictive model, while the other purely technical studies (n = 3) used inferential statistics. The primary categorisation of the analyses was game actions (n = 5), with one study also investigating technical and tactical elements of game styles. Dynamic game actions, collective team behaviours, social network analysis, and movement patterns were not addressed in any of the included studies.

Critical appraisal of eligible studies

The critical appraisal scores are shown in Table 5. Eleven studies were deemed to have a low risk of bias, while one study was deemed to be of a satisfactory risk (Thornton et al., 2020b). Cohen’s Kappa analysis found a fair level of agreement between the two reviewers (κ = 0.334). Full agreement was achieved between reviewers on critical appraisal scores after further consultation. The most notable methodological issue was regarding the representativeness of the data sample to the population (Question 2.3), with a consistent theme of data samples not being representative of the overall population expressed in the aim of the studies. Multiple studies also failed to meet the criteria of how the study size was arrived at (Question 2.4) and the explicit criterion for inclusion of participants (Question 2.1).

Methods of match-play physical performance

The most prominent area of inquiry was physical performance, with all eligible studies in this area investigating running demands, as represented by GPS data. Physical performance research, despite its relative prominence, is not exhaustive, with talent identification, acceleration, training, match outcome, player quality, and physical qualities of strength, speed, and acceleration only assessed in one or two studies. Additionally, contact demands, match importance, agility, and anthropometry were not investigated in the current literature. Notably, in identified studies there is a lack of longitudinal data in all physical performance areas. As such, understanding physical performance analysis across key themes in women’s AF is still in its infancy with many themes unexplored. Explored themes also have greater capacity for re-investigation using current research designs with the addition of more robust samples. It is noteworthy that all physical qualities investigations that involved measures from field-based testing as a factor potentially influencing match-play performance, have all been conducted only on a sub-elite level. This represents a major literature gap with scope for tests of aerobic fitness (e.g.: Yo-Yo tests), speed and acceleration (e.g.: short sprint tests), agility (e.g.: AFL standard agility test), strength (e.g.: relevant repetition maximum lifts), and anthropometry measures (Johnston et al., 2018) all able to be evaluated for a relationship to match-play performance in the elite AFLW competition. Identification of such relationships would provide important data for sport scientists and strength and conditioning coaches, informing what constitutes more appropriate physical fitness assessments and overall training prescriptions.

Data synthesis indicated a discrepancy exists in the definition of high-, very high-speed running (HSR & VHSR) and sprinting measures across the studies. Defining the measures at different thresholds for HSR, VHSR, and sprinting may confuse the interpretation of results and alter the statistical significance of running demands to match-play performance. While two studies (Black et al., 2018a; Clarke et al., 2018) cited the work of Bradley & Vescovi (2015) as the basis for their definition, the initial study’s definition was presented as a range rather than a specific threshold, leading to a discrepancy between the two interpretations. In future research, an exact set threshold and clear definition specific to women’s AF is needed to ensure generalisability and enable a better comparison of results across studies.

Physical running performance data collection for all studies utilised the Catapult Sports Optimeye S5 unit (Catapult Sports, Victoria, Australia) sampling at 10 Hz. This has previously been deemed to be reliable for this purpose and superior to collection through 1 and 5 Hz units (Johnston et al., 2014). Some studies also chose to report other measures of reliability including mean horizontal dilution of precision (HDOP) and mean satellite availability, as well as reporting common practices like that of using the same unit per player across seasons to reduce inter unit error (Clarke et al., 2018; Clarke, Whitaker & Sullivan, 2021; Thornton et al., 2020a). Reporting of these metrics and following these collection protocols should continue across all future studies with only four papers not reporting these metrics (Black et al., 2018a; Black et al., 2018b; Black et al., 2018c; Black et al., 2019b), although all acknowledged the importance of collecting reliable and valid GPS data.

All studies investigating physical performance used traditional statistical tests of significance. The prevalence of this testing method can be somewhat limiting in both conclusions able to be drawn and on the singular perspective of statistical significance that it provides, depending on the data available (Lord et al., 2020). It was also observed that all studies of physical performance utilised effect sizes within their statistical analysis. While effect sizes as a means of interpreting magnitude allows for a more nuanced understanding, issue can be taken on testing conducted without sufficient data or background context that is required in a sport where player and team action is highly conditional on surrounding match context. Using traditional statistical tests in isolation does not fully consider the multi-factorial aspects of match-play, leaving inter-relationships between physical performance metrics uncaptured. Linear cause–effect relationships, while having a place for investigation in the literature as a basis, need to be expanded upon to begin to fully grasp the intricacies involved in match-play performance.

Performance analysis literature has shifted to present match-play in team-invasion sports as a set of ‘dynamic systems’ represented by the constraints or context surrounding each action (Travassos et al., 2013). While confounding factors were addressed through positional consideration in most included studies examining physical performance, the extent to which confounding factors have been captured in data is an issue. The incorporation of further confounding factors, such as weather, opposition strength, phase of play, timing of action, interaction with opposition, and position on the field, is needed to better represent the true running demands of match-play (Travassos et al., 2013). For example, significant differences were found between match phases of play (e.g.: team with or without the ball, stoppage of play) across high- and low-speed running, and acceleration in the AFL (Rennie et al., 2020), suggesting that the phase of play is a key contextual consideration that may be applicable to the analysis of physical performance measures within women’s AF. The further collection and incorporation of data capturing the effects of these contextual factors presents a logical next step for further exploration of existing themes and studies for women’s AF. Doing so would provide justification as to why a player has produced a physical output, as well as mitigate confounding relationships that may lead to inappropriate performance practices being implemented.

Overall, future research from a physical perspective should investigate physical performance over a longer period, mapping the potential development of AFLW athletes as more time is spent under an elite sport program. No testing of the link between physical qualities of players (e.g., aerobic fitness, speed and acceleration, agility, strength and power, and anthropometry) and match-play performance has been conducted at the elite level representing a key area that could inform improved training design and preparation practices.

Methods of match-play technical performance

Of the five studies to address technical skills, reporting effect sizes between playing positions were the method of choice for two studies (Clarke et al., 2018; Clarke, Whitaker & Sullivan, 2021) and reporting of p-value significance the choice of another (Dwyer, Di Domenico & Young, 2022). In contrast, the other two studies used predictive model methods relating match statistics of players and teams to success within games and seasons (Black et al., 2019a; Cust et al., 2019). As an exploratory examination of the technical factors deterministic of match success, the predictive studies are reasonable to use as an initial indication of important game actions within women’s AF. Despite this, there is a lack of appropriate provisions within modelling practices to make declarations of relationships with any certainty or be reflective of anything more than a snapshot in time in a fast-developing environment. This is largely symptomatic of the scarcity in length and contextual information of technical actions within available data applied currently. In the modelling of Black et al. (2019a), results were not verified on a holdout test set or cross-validated, a standard practice in modelling research (James et al., 2013). Potential flaws in the variable selection of Black et al. (2019a), as highlighted by Cust et al. (2019), led to models built with an inherent bias using data that can be seen as a direct determinant of match success. Repeating similar methodology with the removal of these highly predictive variables over multiple seasons of match-play would enable greater robustness and validity of future findings.

The application of generalised estimating equations within Cust et al. (2019) was also not verified on test data, unlike the decision trees in the same analysis. Consequently, caution should be used when drawing conclusions about technical relationships to inform current match-play performance. Match-play technical skills may be considered underdeveloped within the first seasons of the competition. These were the seasons of focus in all previous studies that model technical data within match-play, a further important consideration when interpreting past results for application on the current AFLW competition. Future research re-evaluating previous technical relationships should be sought, aiming to capture and understand the development of the competition since inception. The methodology used should also be carefully considered to enable the capture of this development as has been performed in men’s AF literature (Woods, Robertson & Collier, 2017).

The beginnings of this research into the evolution of the AFLW can be seen in Dwyer, Di Domenico & Young (2022) who investigated how some key match statistics may have changed over the initial three years (2017–19) of the competition. In addition, Dwyer, Di Domenico & Young (2022) also made a comparison to the men’s game from a technical standpoint scaled by the time of play for each competition. While this represents a means of fairer comparison, total match times still do not account for differences in ‘time on’ rules leading to less time in play in the AFLW that will effect comparison in this manner (Australian Football League Women’s, 2021). Twenty-seven variables were analysed for their correlation to match success in the form of match margin, six of which are functions of scoring, meaning that twenty-one unbiased statistics were tested. The significance testing of the correlations of the twenty-one statistics to match margin used is a univariate investigation of variable importance, meaning that results should be taken with caution as many confounding factors could potentially affect this significance.

The influence of confounding factors, such as opposition played, variation of key statistics over time, and team strategy should be considered in these results as well as in any future longitudinal study design to produce a better representation of the key variables affecting match-play performance. More intensive methodologies, including machine learning, can be implemented in the future when appropriate to assist in the incorporation of this additional data. While captured in aggregate form in these analyses (Black et al., 2019a; Cust et al., 2019), each team’s game style may alter which game actions are prioritised to achieve match success. As a result, future research helping to quantify game styles can allow improved interpretation of important technical actions representative of these game styles.

Methods of match-play tactical performance

Given the dearth of tactical performance analysis in the women’s AF literature, there is a large amount of room for exploration within this area. While all of the six tactical domains of the Lord et al. (2020) framework have been investigated in men’s AF, only investigations of game actions (Black et al., 2019a; Clarke et al., 2018; Clarke, Whitaker & Sullivan, 2021; Dwyer, Di Domenico & Young, 2022) and game styles (Cust et al., 2019) have been conducted in women’s AF. All six themes being applied in men’s AF suggests that the ‘dynamic system’ of AF gameplay allows for methodologies in each of these themes to be investigated in women’s AF. Whether they are capable of application in the current women’s AF environment is another question, due to current differences in access to high granularity data available between men and women’s AF. As indicated by observations surrounding the primary game actions studies (Black et al., 2019a; Cust et al., 2019), the theme has scope for greater exploration by considering the variability introduced by the competition’s development and contextual factors on performance.

Cust et al. (2019) can be seen as the beginnings of the tactical investigation of the women’s game, as the study sought to address the strategies employed by a team that results in success. The conclusion that the performance of the highest statistic accumulating players in a range of measures is paramount to team success is a tactical insight, although it does need further context and studies to verify this claim fully. A precedent that has been set for greater investigation of this tactical output within the men’s sphere, is through network analysis utilising more granular data, allowing the mapping of player passing within a team via who is passing and receiving the ball (Young et al., 2020).

Other themes in the tactical categorisations of men’s AF including movement patterns (Sheehan et al., 2021), collective team behaviour (Alexander et al., 2019), social network analysis (Young et al., 2020), and holistic mapping of game styles (Woods, Robertson & Collier, 2017), have not been examined within women’s AF. The first three of these themes require the notation or collection of additional data beyond what is currently available to researchers of women’s AF. Movement patterns and social network analysis would require the availability of spatio-temporal, notated match statistics to capture tactical match-play information; data that are currently captured and used in the AFL (Spencer et al., 2019; Young et al., 2019a) but unused in women’s AF literature. This is possibly due to such data being unavailable in the AFLW or, if available, being beyond the means of many clubs to collect or purchase. Collective team behaviour requires GPS coordinates of all players on a team or whole game to be represented and has been conducted on a proof-of-concept basis in men’s AF (Alexander et al., 2019; Spencer et al., 2019). This may be beyond the reach of current facility constraints of the AFLW, given the requirement of non-aggregated, dynamic game action data being available, particularly with more foundational objectives still unexplored (Alexander et al., 2019). Therefore, with current data provisions in mind, further research of game actions to reflect the development of match-play with additional context and an investigation of game styles should be prioritised. Strides should be made by both clubs, by allocating more resources to self-collection, and the league, by replicating AFL data collection in the AFLW, equalising the availability of data.

Due to the nature of tactical analysis, a prohibitive limitation of studies is that the most common form of data available, aggregate statistics, do not possess the granularity to give greater insights into match-play. It is, therefore, understandable that studies have been limited in scope by the data that is currently available due to the recency of the AFLW competition. Further exacerbating these limitations are the relatively short seasons, and scarce analysis of multiple years of data in current studies. Additionally, the onus of research would be to produce more exploratory studies to give quicker, interpretable insights given the newness of competition, meaning that further data generation processes required to be implemented by teams or the competition have not been prioritised. Future research should seek to implement a data collection protocol that captures game actions with greater granularity, as well as greater contextual information surrounding each action. Specifically, future research is needed to determine what contextual information is important to capture.

The availability and granularity of data remain the greatest limitations within both technical and tactical analysis, which ultimately dictates whether intensive methodologies employed in men’s AF and other sports can be employed for future research in women’s AF. While most applicable to tactical analysis, the greater inclusion of contextual factors that determine match-play phenomena is a gap in the majority of the literature, as player positional consideration has been the only external factor considered in most cases (McGarry, 2009). In a holistic sense, the constraints that each sub-category (physical, technical, tactical performance) imposes upon the other can be incorporated to create more comprehensive models that better reflect match-play circumstances and the influence the three areas have on each other. Besides the natural overlap of technical and tactical performance modelling, no research has yet combined all elements to give a better representation of the multifaceted nature of player and team performance. Precedents exist that enable this information to be incorporated, such as neural networks used by Watson et al. (2020) that can allow for the data from physical, technical, and tactical performance to be utilised in one model allowing intertwined relationships of match-play performance demands to be reflected. Use of these models are still distant, given the intensive data required to perform this modelling in a robust manner. Future research could expand to begin to follow the precedent of the ecological dynamics approach of representing the complex system of match-play performance, while other constraints on match-play performance, such as what a player perceives in a moment in time, can also begin to be investigated. This approach can give the best representation of the requirements for performance and produce more representative training designs for better preparation (Passos, Araújo & Volossovitch, 2017; Vilar et al., 2012).

Further data generation to enable future research should also be considered by the league and teams. However, given current limitations in facilities in the competition, it may not be an immediate focus with current data able to be further utilised. Greater resources allocated by teams toward data collection as well as an equalisation of data availability in the AFLW relative to the AFL either through the primary collector, Champion Data or the league itself, should be sought. All technical and tactical data utilised in current research has been from the Champion Data collection service, which has previously been verified in its reliability and accuracy (Robertson, Back & Bartlett, 2016).

On the objective of greater use of current data, there is still room for more studies (that can still use traditional statistical hypothesis testing) as relationships established from these analyses can dictate the objectives of further research. An example of this is that more Inside 50s has been shown to lead to team success in current literature (Black et al., 2019a). In this case, a more process-oriented approach (Passos, Araújo & Volossovitch, 2017; Vilar et al., 2012) can be sought through analysis of how the Inside 50 statistic is attained in match-play, rather than overall match success being used as the performance outcome objective which can miss key details and trends.

Technical and tactical match-play performance remains only measured in relatively simplistic research designs, with all themes of investigation (game actions, dynamic game actions, movement patterns, social network analysis, collective team behaviours, game styles) able to be either researched for the first time or done with a greater longitudinal data sample that captures more match-play contextual data. Data collection and availability are still potentially prohibitive of to what extent this can be sought in the short term, but efforts should be made by the practitioners, AFLW clubs, and the league to ensure this issue is alleviated in the near future. Upon this data becoming more readily available, more intensive methodologies, including machine learning, will be appropriate and able to give further insight.

Combining physical, technical, and tactical match-play performance measures in future research may provide a more holistic representation of performance, in line with the ‘dynamic systems’ approach of performance analysis. Utilising methods that can incorporate all of this information, like neural networks used by Watson et al. (2020), can allow for this representation to become closer to reflecting intertwined aspects of match-play performance demands.

Critical appraisal

While all but one of the included studies were considered to have a low risk of bias (Thornton et al., 2020b), it is important to note the common methodological issue apparent concerning data representativeness. Consistent issues regarding sample sizes and representativeness of results were evident, primarily due to the short time frame of data available. Only three included studies had inter-seasonal data, and five included studies investigated the data of multiple teams, only two of which were on an elite population (Cust et al., 2019; Dwyer, Di Domenico & Young, 2022). Any analysis of an elite population in women’s AF is naturally constricted by only a few relatively short seasons worth of data due to the recent establishment of the AFLW competition, as well as the COVID-19 pandemic causing an unfinished season in 2020 (Black et al., 2019b; Kleyn, 2021). Exacerbating this issue, the likely evolution of match-play over time further contributes to potential problems in research generalisability. This is in addition to the common constraint associated with team sports, where a population of only one team is often used in research because the directly competitive environment is unconducive to collaboration (McGarry, 2009).

This issue of data collection from one team is more pertinent in physical performance research. Tactical and technical variables investigated have the advantage within the AFL and AFLW of a competition-wide match-play collection service in Champion Data, compared to measures of physical performance that are collected by teams on an internal basis, as evident by the population size of the studies of Cust et al. (2019) and Black et al. (2019a) which incorporate all matches played across multiple seasons.

While much of this review does call for increased data utilisation and collection, caution must be made on the extent of further data collection. A recent report has highlighted issues concerning the ethics and current practice of data collection for performance analysis in sport, with a noted topic being the sensitivity of data collected on players, as well as a suggestion that data that is collected should be only what is needed as dictated by prior evidence (Australian Academy of Science, 2022). The question remains in women’s AF, and much of women’s sport in general, as to what data needs to be collected; it is essential to note that while data requirements can change over time, simply stopping the collection of specific data might hinder future research without a historical time-series of data.

The present scoping review incorporated a systematic search and screening approach to identify available literature to describe methods of match-play performance analysis employed in women’s AF. The critical appraisal of methodological quality performed on included studies strengthened synthesised conclusions regarding match-play performance methodology. Table 6 provides a summary of the key issues and recommendations presented in the discussion of this review. The recommendations provided can assist researchers, sports science practitioners, and data analysts to work towards enhancing performance analysis methods to better understand the physical, technical, and tactical match-play performance in women’s AF