Healthcare applications of single camera markerless motion capture: a scoping review

Review protocol

This review was conducted and reported according to the PRISMA Extension for Scoping Reviews (PRISMA-ScR) (Tricco et al., 2018).

Search strategy

A search was conducted in Pubmed, CINAHL and SPORTDiscus. Figure 1 summarises the stages of article identification, screening, eligibility, and inclusion.

Studies were collected up to 14 January 2022. The following medical Subject headings (Mesh) and key words were used: (application[Title/Abstract] OR software[MeSH] OR system[Title/Abstract] OR deeplabcut[Title/Abstract] OR openpose[Title/Abstract] OR neural network*[title/abstract]) AND (capture[Title/Abstract] OR camera[Title/Abstract] OR mls[Title/Abstract] OR kinect[Title/Abstract] OR 2d[Title/Abstract] OR 3d[Title/Abstract] OR rgb-d camera[Title/Abstract] OR pose estimation[Title/Abstract] OR movement[MeSH] OR motion[MeSH] OR marker*[Title/Abstract]) AND (markerless[Title/Abstract] OR marker-less[Title/Abstract] OR ”single-camera”[Title/Abstract] OR ”single camera”[Title/Abstract]).

Inclusion criteria

Healthcare applications of SCMoCap was the focus of this study. Articles were included that involved the design, development, evaluation or use of SCMoCap systems for an explicitly identified patient group. All quantitative study designs were included in this review. Only studies published in English were considered.

Exclusion criteria

Articles that focussed on animals, involved more than one camera, did not have an explicit stated use for healthcare or did not use markerless motion capture were excluded from this study. Articles that did not include the intended target population in the study were also excluded.

Screening and article inclusion

Two authors (BS and MS) performed screening of title and abstract, and full text before data extraction. Disagreements in inclusion were resolved through discussion with a third author (FP).

Data extraction

A total of 987 articles were identified after duplicate exclusion for the evaluation of title and abstract. A total of 197 articles were selected for full text examination. The final number of articles used in the data extraction was 50.

Table S1 details movement parameters collected in detail.

To address the scoping review aims, data was grouped according to the patient populations evaluated. A description of the single camera system, sample size and patient demographics was included. Any clinical outcome measures used and biomechanical data extracted from the SCMoCap systems was mapped according to The International Classification of Functioning, Disability and Health Framework (ICF) (World Health Organization, 2001). Additionally, any reported limitations, methods of evaluation and measurement errors were evaluated.

Patient group classifications

Patient groups were classified according to the following headings: Neurological Injury in Children; Neurological Injury in Adults; Hereditary/Genetic Neuromuscular Disorders; Frailty; and Orthopaedic or Musculoskeletal. A detailed description of study populations and groupings can be seen in Table 1.

Neurological injuries in children

Neurological injuries in children e.g., cerebral palsy, usually result from an injury to the brain and can occur prior to, during, or after birth, in the early years of life. They comprise a range of permanent sensory-movement disorders, and as a result, movement, balance and posture can be affected, with developmental delays noted in some cases (Rosenbaum et al., 2007; Krägeloh-Mann & Cans, 2009).

This is the only patient group out of the five identified in the search, for whom motion capture is routinely used. Typically consisting of 3D marker-based motion capture of gait combined with EMG recordings, the results are evaluated by a multi-disciplinary team, prior to deciding on a clinical intervention or decision such as surgery (Arnold et al., 2006; Laracca et al., 2014). This type of motion capture is typically applied to children of 5 years and above.

Of the studies identified using SCMoCap, children with cerebral palsy were investigated in two studies: one involving gait (Kidziński et al., 2020) and one involving an upper limb rehabilitation game (Chang, Han & Tsai, 2013). Kidziński et al. (2020) used a regular video camera and a custom OpenPose workflow to track full body keypoints of the limbs, torso and head over time to calculate kinematic clinical outcome measures typically used in practice for treatment planning and monitoring. Outcomes included walking speed, cadence, joint angles, and predicted scores on Motor Function Classification (GMFCS) (Palisano et al., 1997) and Gait Deviation Index (GDI) (Schwartz & Rozumalski, 2008). Kinematics were found to have high correlation with ground truth marker data. GMFCS and GDI predictions were compared against clinical assessment and accuracy for these outcomes were reported to be close to the theoretical upper limit that exists due to patient stride variability (Chia & Sangeux, 2017).

In infant populations, the aim of movement analysis is to diagnose cerebral palsy by evaluating the absence or abnormality of “general movements”, spontaneous non-goal-oriented movements that are displayed up to 20 weeks post-term, and may be an early indicator of cerebral palsy. These movements are clinically evaluated using General Movement Assessment (GMA). GMA has high sensitivity and specificity values (Seesahai et al., 2021), but may still misclassify patients as it is subjective and relies on user knowledge that is difficult to explain to others. As a result, the same movement can be interpreted differently by two observers. In an attempt to automate the assessment of infant movement, marker-based methods (Meinecke et al., 2006) and accelerometery (Heinze et al., 2010) have been used to evaluate general movements, but these methods require attachment of markers on the limbs, which can interrupt the natural occurrence of such movements (Tsuji, Nakashima & Hayashi, 2020).

SCMoCap offers a good opportunity to provide a similar automated assessment, but without the need to attach additional sensors to the infant, and a number of studies that attempted to do just this were identified in the literature. To assess general movements in infants, movement trajectories, and velocities of the upper and lower limbs, which are the same kinematic features assessed through visual inspection by clinicians in the GMA, were measured. Most studies (n = 9) compared the results of their motion analysis with clinical GMA and found high agreement (Adde et al., 2010; Orlandi et al., 2015; Orlandi et al., 2018; Støen et al., 2017; Marchi, Hakala & Knight, 2019; Schroeder, Hesse & Weinberger, 2020; Tsuji, Nakashima & Hayashi, 2020; Reich et al., 2021; Groos et al., 2022). SCMoCap therefore has the potential to assess infant movement and assess gait measurements typically assessed in cerebral palsy populations without interference. Most of the studies identified (n = 8) (Adde et al., 2010; Stahl et al., 2012; Rahmati et al., 2014; Rahmati et al., 2016; Orlandi et al., 2015; Orlandi et al., 2018; Støen et al., 2017; Kidziński et al., 2020; Tsuji, Nakashima & Hayashi, 2020) used a 2D RGB camera to record movement. Using a 2D RGB camera limits the analysis to movements in a single plane, typically frontal for infant studies, but this was not a major limitation as measurements of axial rotation in this population are not a focus. Motion analysis carried out in 2D is also comparable to the approach taken in clinical practice where 2D video is used as part of GMA (Aizawa et al., 2021). Kidziński et al. (2020) demonstrated that accuracy loss that was not a result of patient stride variation could be reduced further by collecting kinematics with input from frontal and transverse planes. However, this is not a single camera solution and serves only to demonstrate the appropriateness of the machine learning model used to recognise movement. Another limitation reported in these studies involved the difficulty in mapping quantitative kinematic outputs from the motion capture measurements to more descriptive characteristics of motion assessed as part of a GMA, such as fluidity and elegance (Orlandi et al., 2015; Tsuji, Nakashima & Hayashi, 2020). This issue would also exist in marker-based systems as these concepts are not easy to objectively quantify with kinematics alone, but a SCMoCap approach offers an opportunity to conduct more research into these higher-level descriptors of movement by enabling more frequent data collection due to reduced setup and portability

Neurological injuries in adults

Neurological injuries in adults can occur as a result of injury or damage to the brain or central nervous system. These can stem from several aetiologies e.g., trauma (Traumatic brain injury), bleeds or blocks to the vascular system of the brain e.g., stroke (Lawrence et al., 2001), autoimmune or immune mediated neural and glial damage e.g., multiple sclerosis (Isaksson, Ahlstrom & Gunnarsson, 2005), or altered neuro-chemical processes e.g., Parkinson’s disease (Dauer & Przedborski, 2003) or epilepsy. Neurological injuries to the brain or central nervous system can affect several body systems such as visual or auditory, but in all cases of neurological injury, voluntary movement is affected.

In this group, SCMoCap was typically used to measure kinematics that occur during observer based ordinal or single measure clinical outcomes, in an effort to provide additional information that may be useful for treatment but is not typically assessed. The Microsoft Kinect v1, a hybrid RGB and infra-red depth sensing camera, was used in ten of the identified studies (Lange et al., 2011; Behrens et al., 2014; Galna et al., 2014; Rocha et al., 2014; Vernon, Paterson & Bower, 2015; Behrens et al., 2016; Grobelny et al., 2017; Bonnechère et al., 2018; Grunert, Krause & Feig, 2019). Kinect v2, featuring a higher resolution, was used in six (Rocha et al., 2015; Ozturk et al., 2016; Eltoukhy et al., 2017; Latorre et al., 2018; Pereira Choupina et al., 2018; Nakamura et al., 2019) and the Leap Motion sensor, a smaller camera with similar technology, was used in four (Chen et al., 2016; Grobelny et al., 2017; Butt et al., 2017; Oña et al., 2018). Two studies used only an RGB camera (Rémi et al., 2011; Sato et al., 2019). Cameras with depth sensors as well as RGB sensors can record movements in 3D and have the potential to provide comparable accuracy to gold standard, 3D marker-based systems. This makes them particularly useful compared to 2D motion capture when dealing with motions where out of plane rotations are important.

In Parkinson’s disease, physiotherapy has been demonstrated to increase functional ability (Tomlinson et al., 2013). In this population, inertial sensors (Giggins, Sweeney & Caulfield, 2014), tablet (Dauvergne et al., 2018), and mouse and keyboard (Van de Weijer et al., 2019) have been used to record motion for serious games for rehabilitation. However, these methods require a level of hand function that is typically only available to patients at an early stage of the disease, potentially excluding patients with more advanced Parkinson’s from participating in the games. Inertial sensors may also be difficult to set up for home rehabilitation. A markerless approach can increase the effectiveness of rehabilitation by facilitating the continuous repetition of exercises at home. Most studies using SCMoCap for Parkinson’s patients aimed to provide biofeedback for rehabilitation games (Palacios-Navarro, García-Magariño & Ramos-Lorente, 2015; Capecci, Ceravolo & Ferracuti, 2018; Oña et al., 2018; Grunert, Krause & Feig, 2019). Oña et al. (2018) reported measuring upper limb joint trajectories and range of motion with a Leap Motion controller to achieve this. The efficacy of the games was tested by demonstrating improvement in lower limb function by completion time in the 10 Meters Walk Test score (Palacios-Navarro, García-Magariño & Ramos-Lorente, 2015); and upper limb function by evaluating performance using Jamar handgrip dynamometer, Box and Blocks Test and Purdue pegboard test (Oña et al., 2018). In this group then, SCMoCap offered real value to the treatment of patients by enabling the capturing of performance data not easily achievable by other means.

SCMoCap was also used in a number of studies to discriminate between Parkinson’s and non-Parkinson’s patients (Galna et al., 2014; Rocha et al., 2014; Rocha et al., 2015; Eltoukhy et al., 2017; Butt et al., 2017; Lee, Sinclair & Jones, 2019) using upper and lower limb kinematic measurements. Upper limb measurements mainly focussed on velocity, acceleration and frequency of the hand and wrist during activities such as supination and pronation; hand opening and closing; and finger tapping (Galna et al., 2014; Butt et al., 2017; Lee, Sinclair & Jones, 2019). Lower limb measurements mainly focussed on collecting acceleration, velocity, distance between legs, range of motion, stride duration and cadence during gait (Galna et al., 2014; Rocha et al., 2014; Rocha et al., 2015; Eltoukhy et al., 2017). Butt et al. (2017), Lee, Sinclair & Jones (2019) compared these measurements against those obtained using the MDS-Unified Parkinson’s Disease Rating Scale (Goetz et al., 2008), a clinical scale used to evaluate Parkinson’s disease severity with items that focus on motor function. Butt et al. (2017) found difficulty recognising forearm supination and pronation, finger trajectories and tremors due to the high speed of the movements, resulting in limited clinical association. The ability to detect bradykinesia, slowness of movement associated with Parkinson’s, had strong agreement with clinical scores using measurements of wrist supination and pronation, and hand opening and closing movements; however, they found that finger tapping contributed little predictive power to the model due to inaccuracies with tracking using a Leap Motion (Lee, Sinclair & Jones, 2019). Galna et al. (2014) compared Kinect measurements against a 3D marker-based system during gait and upper limb reaching and grasping tasks. They found the Kinect to be capable of measuring temporal and gross spatial parameters; however, it lacked the precision to track minor spatial movements of the hands and fingers accurately.

In multiple sclerosis (MS), studies aimed to quantitatively assess functional decline by measuring gait and postural control. In this population, motion capture typically involves GAITRite (CIR Systems Inc. Clifton, NJ, USA), a pressure sensitive walkway that provides spatiotemporal markers of gait (Sosnoff, Sandroff & Motl, 2012), and marker-based systems (Galea et al., 2017). These methods are highly accurate (Shanahan et al., 2018), but they require a laboratory environment and a long setup time, and markers may obstruct natural gait. Several studies investigated the use of SCMoCap to quantify the Short Maximum Speed Walk (SMSW), a new proposed observer independent measure of detecting walking speed and sway using hip joint centre tracking (Behrens et al., 2014; Behrens et al., 2016; Grobelny et al., 2017). They compared SMSW against established observer based clinical measures of gait disability such as 25-foot walk (Polman & Rudick, 2010), where maximum walking speed is assessed over a distance of 25 ft by an observer with a stopwatch and is often used as part of the Multiple Sclerosis Functional Composite (MSFC), and Expanded Disability Status scale (Kurtzke, 1955), a scale used to determine how MS affects the individual by assessing function such as muscle weakness and balance. SMSW was found to be superior to T25FW in the most recent study as it can detect walking speed over one or two strides—making this a more suitable test for those with limited gait function. This is a promising application of SCMoCap, as these measures rely on large-scale planar movements and do not require the accurate measurement of small segment motions or axial rotation of segments. In this scenario, SCMoCap can effectively replace more cumbersome setups such as the walkway.

In stroke patients, inertial (Mizuike, Ohgi & Morita, 2009) and 3D optoelectronic systems (Chen et al., 2021) have been used to assess movement with high accuracy. SCMoCap studies involving people who have had a stroke mainly focused on collecting kinematics that occur during single outcome clinical assessments but are not typically measured, to provide additional kinematic information about performance that could be useful for treatment. The primary focus of these studies was gait (Vernon, Paterson & Bower, 2015) assessments such as 10 meter walk test (10 MWT) and Timed up and go (TUG); and upper limb assessments (Ozturk et al., 2016) such as Wolf Motor Function Test (WMFT), which scores motor ability during functional tasks (Wolf et al., 2005). (Vernon, Paterson & Bower, 2015) collected gait kinematics during a TUG such as range of trunk angle, step length, stride length, cadence and gait speed to predict outcomes on TUG and 10MWT performed by a clinician and found these measurements to accurately predict outcomes. Ozturk et al. (2016) collected upper limb kinematics such as range of motion, speed and joint trajectories with a Kinect during a reaching task and could predict performance on clinical WMFT and discriminate between stroke and healthy controls. Scano et al. (2014) compared upper-limb kinematics collected from a Kinect v1 against a 6 Camera 3D marker-based system (SMART BTC, Italy) during a reaching against gravity task. This was one of the few studies to report joint angle error of measurement in degrees; where shoulder elevation angle had a reported difference of 3.32 degrees ± 2.80 degrees and elbow flexion-extension angle had a reported difference of 5.60 degrees ± 6.35 degrees. Both errors of measurement were reported to be clinically acceptable.

An epilepsy diagnosis depends on electroencephalogram (EEG) readings and characteristics of seizure induced movements. Inertial sensors (Nijsen et al., 2005) and marker-based methods (Cunha et al., 2012) have been used to analyse seizure movements in Epilepsy Monitoring Units, with the advantage of being a quantitative approach to studying movements of interest that were traditionally only assessed by visual inspection from a clinician. However, several impracticalities have resulted from this method: mainly, the difficulty in attaching sensors and markers in a way that they do not detach during seizures, resulting in these methods being unsuitable for violent seizures. One Kinect per patient was used to count and discriminate between seizure types in an Epilepsy Monitoring Unit (EMU) setting in one study (Rémi et al., 2011; Pereira Choupina et al., 2018). Kinematics collected during a seizure included trunk speed, wrist speed and trajectories of trunk center and wrists. The seizure movements demonstrated high correlation with marker-based measurements (Cunha et al., 2016) and seizure classifications were validated by two clinicians (Pereira Choupina et al., 2018).

The benefit of the SCMoCap approach in adults with neurological injury compared to marker-based systems is reduced setup time, and the avoidance of potentially intrusive markers. The main technical limitation reported was accuracy (n = 5), specifically with high frequency movements, such as tremors (Chen et al., 2016; Butt et al., 2017); forearm supination and pronation (Butt et al., 2017); minor movements, such as finger tapping (Lee, Sinclair & Jones, 2019) or hand clasping (Galna et al., 2014); low resolution of video in a study using only an RBG camera (Rémi et al., 2011); and general accuracy of sensor, even with the higher resolution version of the Kinect (Nakamura et al., 2019). Two studies reported having issues with getting patients to perform the correct movements in front of the camera (Butt et al., 2017; Grunert, Krause & Feig, 2019), but this is of course a limitation not unique to SCMoCap technology.

Hereditary/genetic neuromuscular disorders

Hereditary disorders occur as a result of altered genes which can affect several signalling pathways in the body (Mirkin, 2006). These genes can be transmitted between generations and, depending on the pathway affected, result in changes to a range of body systems such as skeletal muscle and peripheral nerves which can affect movement (Conceição et al., 2016; Lim, Nguyen & Yokota, 2020).

SCMoCap in this group was utilised with the intention of monitoring disease progression and planning rehabilitative treatment. All studies used cameras that include depth sensors, three studies used a Kinect v2 (Carmo Vilas-Boas et al., 2019; Vilas-Boas et al., 2019; Vilas-Boas et al., 2020) and two studies used a Kinect v1 (Han et al., 2015; Han, Bie & Nicorici, 2016). Depth sensors were utilised to be comparable to 3D marker-based systems and for studies involving understanding functional workspace (Han et al., 2015; Han, Bie & Nicorici, 2016). A 3D representation of the workspace was required to better track and understand the subject’s movement trajectories within the workspace. Diseases in this group are rare and consequently there is limited kinematic data available on their progression; therefore, SCMoCAP is invaluable in the effort to understanding these diseases as it has the potential to facilitate practical and more frequent monitoring such as in the home of patients.

Transthyretin-Related Familial Amyloid Polyneuropathy (TTR-FAP) is an adult-onset, neurodegenerative disease causing muscular weakness and other systemic issues. Movement analysis in this group aims to support diagnosis and prediction of future impairment. Studies using the Kinect v2 aimed to differentiate asymptomatic, symptomatic and control groups by measuring spatiotemporal and kinematic parameters during gait (Carmo Vilas-Boas et al., 2019; Vilas-Boas et al., 2019). Spatiotemporal measurements of gait calculated included stride and step duration; step and stride length; step and gait speed. Kinematic parameters during gait included joint angles for the spine, elbow, knee and hip. They compared these to marker-based systems and found higher agreement for spatiotemporal features than kinematic. Most spatiotemporal parameters were found to be clinically relevant; however, out of the kinematics studied, only minimum elbow angle was found to be useful for gait assessment.

Limited upper extremity functional assessment methods exist for assessing patients with facioscapulohumeral muscular dystrophy (FSHD) and Duchenne muscular dystrophy (DMD). FSHD evaluation scale (Lamperti et al., 2010) involves scores provided by an observer, that aim to assess upper extremity function and are used to track muscle weakness and functional decline within disease progression. For DMD, Performance of Upper Limb assessment (PUL) (Mayhew et al., 2013) aims to track the progression in muscle weakness and decline of function. FSHD (Han et al., 2015) and DMD (Han, Bie & Nicorici, 2016) studies using SCMoCap aimed to compare reachable workspace as a measure against existing clinical evaluations of upper limb function, to better understand disease progression and monitor recent therapeutic interventions. The reachable workspace was calculated using parameters including movement trajectories and range of motion. FSHD scores were compared against a SCMoCap approach using a Kinect and were found to have high agreement. Reachable workspace showed strong correlation with PUL evaluation of DMD. FSHD and DMD studies reported that the reachable workspace outcome measure can differentiate upper extremity function levels with adequate sensitivity.

The main limitation reported by almost all studies in this group (n = 4) (Han, Bie & Nicorici, 2016; Vilas-Boas et al., 2019; Vilas-Boas et al., 2020) was small sample size. This was further associated with the rarity of the diseases, making recruitment of subjects for various disease stages difficult.

Frailty

Frailty (Xue, 2011; Clegg et al., 2013) is a disorder associated with increased vulnerability, stemming from a decline in multiple body systems. This is associated with poorer health outcomes such as higher falls risk, hospitalisation and mortality and is usually classified according to the presence of phenotypes e.g., slower walking speeds, low physical activity or unintentional weight loss. Frailty is distinct from normal processes associated with healthy ageing; however, some phenotypes used in the diagnosis of frailty may be present in some older adults.

The aim of most studies using SCMoCap in this group was to improve fall risk assessment. Studies measured at-home gait parameters: one study estimated Timed up and go time (TUG) (Stone & Skubic, 2013) which is used as part of a typical fall risk assessment; another used gait data collected to detect falls in the home (Stone & Skubic, 2015). The Kinect was used as a practical method to collect gait sequences in all studies. Other technologies used for fall detection include floor vibration (Alwan et al., 2006), passive infrared (Sixsmith, Johnson & Whatmore, 2005) and acoustic (Zigel, Litvak & Gannot, 2009) sensors. These methods provide mostly spatiotemporal data, whereas the Kinect has the ability to measure further information such as body segment kinematics preceding a fall, that may give insight into the causes of falls and lead to improved methods of falls prevention. Depth information from the Kinect was also utilised as it is robust to variation in ambient lighting and to understand the 3D space in which the fall takes place.

Spatiotemporal parameters of gait were mainly collected, such as speed, stride time and length, and height, over a period of up to 16 months. The SCMoCap studies evaluated the technology using the clinical assessment of TUG against SCMoCap-estimated TUG from in home gait speed (Stone & Skubic, 2013), statistical validation against state-of-the-art fall detection algorithms (Stone & Skubic, 2015) and a comparison against hand labelled data (Stone & Passive, 2012). In home gait speed was found to be accurate at predicting TUG performance. All studies reported success in capturing the majority of natural gait sequences and falls. Stone & Skubic (2015) reported technical limitations with occlusion and issues with limited field of view, requiring falls to be in the view of the sensor which made the capture of some gait sequences not possible. This emphasises the importance of camera positioning in such studies, and suggests that the size of the space being monitored needs to be carefully considered.

Orthopaedic or musculoskeletal

The classification (Punnett & Wegman, 2004) of musculoskeletal or orthopaedic disorders encompasses a broad range of conditions, such as arthritis or tendinitis, with varied disease mechanisms and causes e.g., acute trauma, overuse trauma, degeneration or inflammation. Whilst the skeletal and muscular systems are predominantly affected e.g., joints, muscles, and tendons, it can also include peripheral nerves and associated vascular structures. These conditions can impair movement and can be managed using surgical and non-surgical interventions. SCMoCap in this group was mainly carried out to compare the accuracy of the Kinect with clinical outcome measures. All studies in this group used the Kinect to provide a 3D understanding of movements produced by multiaxial joints such as the hip and shoulder.

One single case study aimed to monitor changes in gait before and after total hip replacement surgery (Dolatabadi, Taati & Mihailidis, 2014) with the intention of understanding balance issues after surgery. Outcome measures assessing balance after surgery are typically in the form of self-report which does not always align with clinical assessment (Gandhi et al., 2009). To achieve this spatiotemporal and kinematic parameters were calculated using a Kinect. Spatiotemporal measurements collected during gait included step length, stance time and cadence. kinematics collected during a sit to stand exercise included hip angular velocities and center of mass. Results obtained were statistically compared against the subjects own self-assessment during rehabilitation and were found to agree well.

Another tested the agreement of Kinect measurements against clinical goniometer readings of upper limb range of motion in patients with adhesive capsulitis of the shoulder (AC); during flexion, abduction and external rotation (Lee, Yoon & Chung, 2015). AC is typically diagnosed based on shoulder range of motion measurements usually with a goniometer. Goniometer measurements are standard in clinical practice; however, results can differ between observers (Riddle, Rothstein & Lamb, 1987). The Kinect was demonstrated by this study to be a clinically viable tool for assessing active and passive shoulder range of motion and diagnosing AC, given the excellent agreement with goniometer readings. Inertial (Mullaney et al., 2010) and 3D marker-based (Ropars et al., 2015) methods have been used to assess shoulder range of motion in the past with excellent accuracy; however, these methods require setup and do not compare with the practicality of the Kinect.