Subject

Gait analysis assesses the gait capabilities of a subject by quantifying kinematics (mostly joint angles) and kinetics (mostly joint moments) parameters. Spatiotemporal parameters (e.g. step length, step time and walking speed) are also popular in the rehabilitation field to assess for example the gait of elderly people and neurological patients (e.g. Stroke and Parkinson Disease). Gait analysis based on marker based mocap such as Vicon - the state of the art - is performed using a biomechanical model, representing the human body as segments linked by balls joint in a chain. The orientation of the segments is determined by multiple cameras tracking the positions of infrared markers placed on the subject body. Marker based systems (e.g. Vicon) are therefore not portable and are difficult to use. Instead, the Kinect RGBD sensor offers a marker less, cheap, portable solution to track the subject body. Nevertheless, Kinect is limited by its inherent inaccuracy and the measured joint angles are generally not enough accurate for clinical assessment.

Kinect performs markerless motion capture by applying to single depth images a Machine Learning (ML) human pose estimation algorithm. The output can be inconsistent between frames (e.g. the lengths of model’s segments could change over time), and be affected by noise. Different approaches have been explored to refine the Kinect estimated pose [1-3], correct unnatural poses and integrate physical constraints in the kinematic model. Promising approaches are based on Recurrent Neural Network (RNN) such as Long short-term memory (LSTM), which are able to make predictions in real time and do not require the time normalization of data.

To accurately evaluate Kinect's tracking performance, is essential to concurrently collect ground truth Vicon data. However, establishing a robust and repeatable protocol presents challenges due to various factors affecting Kinect's skeletal tracking, such as ambient lighting conditions, the sensor's placement (height, distance, and angle relative to the subject), and specific Kinect settings [4]. But the biggest challenge is that the reflective markers significantly disrupt Kinect's depth images, impairing its tracking algorithm [4, 5]. While some studies (e.g. [4]) have opted to assess the systems separately to avoid marker interference, this approach does not allow for a direct comparison. Addressing this issue, we propose employing a deep learning strategy to preprocess and denoise the depth images prior to analysis by the Kinect skeletal tracking model, aiming to mitigate the interference caused by reflective markers and enhance the accuracy of Kinect's tracking capabilities.

Kind of work

The aim of the thesis is to enhance the skeletal tracking capabilities of Kinect by:

defining optimal acquisition settings

Developing DL models to denoise the depth images (preprocessing)

Developing DL models to refine the estimated pose and/or joint angles (postprocessing)

The performances of the DL models will be systematically evaluated using prior datasets collected at the Brubotics Rehabilitation Research Center (BRRC). Datasets contain concurrent acquisitions of Kinect and Vicon data from healthy volunteers. The evaluation of the optimal acquisition settings will be also carried out at BRRC.

The project will consist of:

Literature study

Development and implementation of a protocol to define optimal acquisition settings

Development and implementation of DL models to enhance Kinect skeletal tracking

The methods will be implemented in Python using open-source libraries and common deep learning frameworks. Additionally, basic knowledge of C++ is required to work with Azure Kinect SDK. Prior data collected at BRRCC are available, but a small data acquisition may be needed to define the optimal acquisition settings. Data acquisition will require to operate the Vicon system (calibration, markers placement onto the subject body, use of the Vicon SDK software) and the Kinect system.

Framework of the Thesis

References

[1] Ma, Ye & Liu, Dongwei & Cai, Laisi. Deep Learning-Based Upper Limb Functional Assessment Using a Single Kinect v2 Sensor. Sensors. 2020

[2] Park, Youngbin & Moon, Sungphill & Suh, Il Hong. Tracking Human-like Natural Motion Using Deep Recurrent Neural Networks. 2016

[3] Li, Ruotong & Si, Weixin & Weinmann, Michael & Klein, Reinhard. Constraint-Based Optimized Human Skeleton Extraction from Single-Depth Camera. Sensors. 2016

[4] Yeung LF, Yang Z, Cheng KC, Du D, Tong RK. Effects of camera viewing angles on tracking kinematic gait patterns using Azure Kinect, Kinect v2 and Orbbec Astra Pro v2. Gait Posture. 2021

[5] Y. Ma, B. Sheng, R. Hart and Y. Zhang. The validity of a dual Azure Kinect-based motion capture system for gait analysis: a preliminary study. APSIPA ASC. 2020

Number of Students

1