Technical quality assessment of an optoelectronic system for movement analysis

The Optoelectronic Systems (OS) are largely used in gait analysis to evaluate the motor performances of healthy subjects and patients. The accuracy of marker trajectories reconstruction depends on several aspects: the number of cameras, the dimension and position of the calibration volume, and the chosen calibration procedure. In this paper we propose a methodology to evaluate the effects of the mentioned sources of error on the reconstruction of marker trajectories. The novel contribution of the present work consists in the dimension of the tested calibration volumes, which is comparable with the ones normally used in gait analysis; in addition, to simulate trajectories during clinical gait analysis, we provide non-default paths for markers as inputs. Several calibration procedures are implemented and the same trial is processed with each calibration file, also considering different cameras configurations. The RMSEs between the measured trajectories and the optimal ones are calculated for each comparison. To investigate the significant differences between the computed indices, an ANOVA analysis is implemented. The RMSE is sensible to the variations of the considered calibration volume and the camera configurations and it is always inferior to 43 mm.


Introduction
The Optoelectronic Systems (OS) are largely used in gait analysis to evaluate the motor performances in clinical analysis [1]. The accuracy of tracking marker trajectories is dependent on several aspects: (i) the number and position of the cameras [2], (ii) the dimension and position of the calibration volume [3,4], (iii) the calibration procedure [5,6] and (iv) the algorithm which converts the 2D coordinates of markers acquired by each camera in the 3D space [7]. The reconstruction phase is performed by OS software, therefore it is not possible to specifically evaluate the effect of the implemented algorithm and the other causes on the interval of uncertainty associated to the measurements. In literature several methods to evaluate the accuracy of OS are proposed. Weng et al. [6] designed a novel static calibration procedure; they imposed known static input to the OS by means of a custom-made calibration stand. Then, the actual positions of a set of points on the calibration stand (addressed as testing points) are compared with the measured position. Vander Linden et al. [3] calculated the error on marker position similarly to Weng et al. [6] by testing the accuracy of the OS in nine points of the calibration volume (54.3 × 72.0 × 42.3 cm 3 ); Branca and Cappa [4] evaluate the metrological performances via an ad-hoc system capable to impose known marker trajectories. Ehara et al. [1] provide an estimation of the accuracy on the dynamic reconstruction, comparing the actual value of the distance between two markers on a rigid wand and the values calculated by the OS while moving the wand in the capture volume. In order to provide repeatable trajectories of markers as input for the OS, Windolf et al. [2] proposed a xyz-robot which moved a L-wand equipped with reflective markers inside a volume of 18.0 × 18.0 × 15.0 cm 3 . In this paper we propose a novel methodology to evaluate the accuracy of the OS. We consider all the above reported causes of uncertainties, enlarging the tested volume to a value comparable with the ones normally used in clinical gait analysis.

Methods
The optoelectronic system analyzed was the Vicon MX (Oxford Metrics) composed by eight IR cameras. The OS is installed at the Movement Analysis and Robotics Laboratory of the Children Hospital 'Bambino Gesu' (Palidoro, Rome IT). The reconstruction of marker trajectories was conducted by means of the OS manufacturer software. The experimental protocol consists of a calibration, validation and post-processing phases.
In the calibration session, as also proposed by Vander Linden et al. [3], we divided the capture volume (2.4 × 3.6 × 1.6 m 3 ) into 12 sub-volumes (1.2 × 1.2 × 0.8 m 3 ), six at the floor level and six at a higher level (see Figure 1). The calibration procedure was performed moving a 5-markers wand into each selected sub-volume. Each calibration procedure was acquired at 200 Hz for 15 s at least, and it was automatically stopped after each camera has acquired at least 5,000 frames. At the end of the calibration session 13 calibration files have been acquired: 12 for each sub-volume and one for the overall volume. In the subsequent validation session, the tip of 1 m length wand was equipped with a reflective marker. The operator moved it into the overall volume. Finally, in the post-processing phase, it was possible to apply to each trial acquired in the validation session the 13 calibration files acquired during the calibration phase. In order to evaluate the influence of the redundancy of the camera numbers, we decided to consider five different camera configurations: the first one with all the eight cameras (addressed as all-camera test) and the other four with only three cameras switched on at the same time (addressed as 3-camera test). Even though the minimum number of cameras necessary to track a marker trajectory is, as well known, equal to two we decided to limit it to three since it is the value provided by the manufacturer as optimal value. The accuracy of the OS was evaluated by means of two indices. The first index used in the present study is the RMSE C, i.e. the RMS of the difference between the i -th trajectory tracked using one of the '3-camera' configurations and the optimal one ('all-camera') assumed as reference; this index quantifies the effect of the camera redundancy. The second index is the RMSE V, which gives the effect of the variation of the calibration volume. It is evaluated as the RMS of the distance between the i -th trajectory processed with one of the 12 sub-volume calibration files and the trajectory processed with the file processed taking into account the whole volume calibration.
Significant differences in the indices were evaluated by means of two-way ANOVA (13 × 5, i.e. volumes × camera configurations) (p=0.05). Tukey tests were considered as post-hoc test with the same alpha level.

Results and Discussion
Considering the whole calibration volume, the mean value of the reconstruction error due to different camera configurations was estimated as equal to 3 mm (max value 9 mm). Considering different volumes with 'all-camera' configuration, the mean value of the reconstruction error was 17 mm (max value 43 mm). The Figure 2 shows the marker trajectory reconstructed by using one of the '3-camera' configuration and four different calibration volumes, those are highlighted in the figures by a cube. Instead, the Figure 3 shows the reconstructed trajectory by using one of the sub-volume calibration files and four different '3-camera' configurations. The marker size in the pictures is equal to the distance of the trajectory from the optimal one (that is the one tracked by using the all-camera configuration in the case of Figure 2 and the whole calibration volume for the Figure 3). Changing the camera configuration or the calibration volume, the reconstructed trajectory can be unavailable in any parts of it because of the algorithm, which compute the reconstruction. This is not known due to the manufacturer, but probably the unavailability of the trajectories occurs because the algorithm does not find a unique solution for the marker positioning at those frames. In fact, the calibration parameters (i.e. the positions of each camera relative to each other and to the absolute reference frame) are strongly dependent on the calibration volume; therefore the reconstruction of marker trajectory is up to the volume as well.
Unlike what we expected, applying the calibration file related to a specific sub-volume, the reconstruction of the trajectory that is performed in this volume is not more accurate than the one conducted in the other volumes. Both in the cases of RMSE C and RMSE V the two main effects give significant differences (p < 0.01). Significant differences are always found between sub-volumes and whole calibration volume (p < 0.01) and between the 'all cameras' configuration and the other '3-camera' tests (p < 0.01). Thus, the calibration volume has to be close to the chosen measurement volume; in fact, changing the dimension of calibration volume, the RMSE C decrease from a maximum mean value of 4 mm to 1 mm, and RMSE V decreases from a maximum mean value of 30 mm to 4 mm. The obtained results not only confirm the already stated relationship between the number of cameras and system accuracy, but also allowed a further quantification of the previously mentioned relationship. In fact, considering the 'allcamera' configuration, in the central sub-volumes the RMSE V decrease from a maximum of 30 mm on bounds of the whole calibration volume to a minimum value of 6 mm. The Table 1 shows the RMSE C values which are related to the number of cameras used for the reconstruction of the trajectory when the whole calibration volume is considered. As expected,  well. By using six cameras or seven, the RMSE does not change significantly.

Conclusion
In the present study we evaluated the accuracy of the optoelectronic system keeping into account the main causes of inaccuracies. From the obtained results, we can assert and quantify the relationship between overall accuracy of the optoelectronic system and the selected calibration volume together with the number of used cameras.