Surgical navigation for guidewire placement from intraoperative fluoroscopy in orthopaedic surgery

Objective. Surgical guidewires are commonly used in placing fixation implants to stabilize fractures. Accurate positioning of these instruments is challenged by difficulties in 3D reckoning from 2D fluoroscopy. This work aims to enhance the accuracy and reduce exposure times by providing 3D navigation for guidewire placement from as little as two fluoroscopic images. Approach. Our approach combines machine learning-based segmentation with the geometric model of the imager to determine the 3D poses of guidewires. Instrument tips are encoded as individual keypoints, and the segmentation masks are processed to estimate the trajectory. Correspondence between detections in multiple views is established using the pre-calibrated system geometry, and the corresponding features are backprojected to obtain the 3D pose. Guidewire 3D directions were computed using both an analytical and an optimization-based method. The complete approach was evaluated in cadaveric specimens with respect to potential confounding effects from the imaging geometry and radiographic scene clutter due to other instruments. Main results. The detection network identified the guidewire tips within 2.2 mm and guidewire directions within 1.1°, in 2D detector coordinates. Feature correspondence rejected false detections, particularly in images with other instruments, to achieve 83% precision and 90% recall. Estimating the 3D direction via numerical optimization showed added robustness to guidewires aligned with the gantry rotation plane. Guidewire tips and directions were localized in 3D world coordinates with a median accuracy of 1.8 mm and 2.7°, respectively. Significance. The paper reports a new method for automatic 2D detection and 3D localization of guidewires from pairs of fluoroscopic images. Localized guidewires can be virtually overlaid on the patient’s pre-operative 3D scan during the intervention. Accurate pose determination for multiple guidewires from two images offers to reduce radiation dose by minimizing the need for repeated imaging and provides quantitative feedback prior to implant placement.


Introduction
Surgical guidewires, such as Kirschner wires (K-wires), help guide placement of fixation implants (e.g.screws, plates) that are commonly used for stabilizing fractured bones in orthopaedic trauma surgery (Franssen et al 2010, Kim et al 2020).Such procedures require extensive use of fluoroscopy, for example, using a mobile C-arm to visualize the instruments in relation to the treated anatomy.Studies show that trauma surgeons are exposed to higher dose levels than other orthopaedic subdisciplines-a year-long study by Gausden et al (2017) showed mean exposure of 530 μSv/month.Despite extensive use of imaging, accurate 3D positioning of guidewires remains a challenge.Reviewing 103 pediatric fracture fixation cases, (Sharma et al 2007) measured a 32.3% complication rate associated with K-wire placement (e.g.wire loosening due to malplacement).Limited accuracy, the potential for injury, and high levels of radiation exposure motivate the use of 3D navigation for guidewire placement.
Surgical navigation using trackers, while more prevalent in cranial and spinal neurosurgery, show limited adoption in orthopaedic trauma.A major challenge is tracking fractured anatomy; dedicated reference markers for each bone fragment are not feasible using conventional modalities.Optical/infrared trackers require an unobstructed line-of-sight that is particularly difficult when considering the variations in patient positioning and screw trajectories.Electromagnetic tracking suffers from field distortions by metal objects, including most of the instruments and implants used in current practice (Leung et al 2010).These limitations and the existing use of imaging motivate the development of image-based solutions that do not require additional tracking equipment or custom markers.
A large body of prior work on image-based guidewire navigation relies on physical modeling of instruments.The method by Cleary et al (2003) segments surgical needles from fluoroscopy by exploiting their tube-like characteristics using a series of Gaussian filters and gradient computations.Prior work by our group defined 3D flexible (Uneri et al 2016(Uneri et al , 2021) ) and multi-part (Uneri et al 2015) component models for use in 3D-2D image registration to fluoroscopic images.While the accuracy and precision of these approaches are well established, iterative numerical optimization can result in long runtimes that may not be well suited for a rapid surgical workflow.
Machine learning approaches using convolutional neural networks (CNN) have been used in a wide variety of 2D radiographic detection and segmentation tasks, including orthopaedic trauma applications.The method by Zhou et al (2020) uses a pyramid recurrent attention network for automatic segmentation of guidewires in fluoroscopic images.Doerr et al (2020) used Faster R-CNN for detecting spinal pedicle screws from cone-beam CT (CBCT) projections.Cho et al (2021) used a custom CNN to detect surgical instruments' tips in biportal endoscopic spine surgery.
Learning-based 2D detection approaches have also been used in estimating 3D poses of instruments.The method by Esfandiari et al (2018) performs multi-class semantic segmentation of pedicle screw heads and shafts from biplanar x-ray projections and estimates the 3D screw poses by intersecting corresponding central axes.Correspondence between different screw segmentations is formed using the central screw axis in the first view to estimate the epipolar line in the second view.Hasan et al (2021) trained a CNN to detect, segment, and extract surgical instruments' edge, mid-line, and tip from laparoscopic images.The 3D poses of instruments were estimated from these image features by modeling a tool as a 3D cylindrical shaft with an arbitrarily shaped head.von Atzigen et al (2021) used a CNN for detecting screw heads in video frames from two stereo cameras.Correspondence of the detections from each view was then established by assuming a pure translation between the two stereo cameras (i.e.neglecting the rotational component).This assumption reduced the correspondence problem to finding vector pairs formed by detections from each view that point in the same direction.
This paper presents the development and preclinical evaluation of an algorithm for the 3D navigation of orthopaedic guidewires from intraoperative fluoroscopy.The approach consists of: (1) detection of guidewire tips and directions from a pair of images; (2) 3D backprojection of detected guidewire tips; and (3) correspondence of detections from multiple views to identify the 3D poses of guidewires.Motivated by the need to reduce radiation exposure, the intended use case is a step-and-shoot fluoroscopy workflow (see continuous fluoroscopy), in which single 2D projections are acquired to provide near-real-time guidance.While methods for 3D guidance of surgical instruments are often either model-or learning-based, the proposed method leverages the benefits of both solutions by combining rapid 2D detection with robust model-based 3D localization and correspondence.The presented approach builds on preliminary work by Bataeva et al (2021).Contributions include a combined network architecture for the simultaneous detection of guidewire tip and direction in detector coordinates, a model-based localization and correspondence method to estimate a guidewire's tip in world coordinates, and an iterative 3D direction method developed to account for guidewires located in the imaging plane.

Methods
The approach for 3D guidewire navigation from 2D fluoroscopic images, illustrated in figure 1, combines datadriven object detection with model-based localization and correspondence.Pairs of intraoperative C-arm images at different gantry tilts are provided as input.A region-based CNN (R-CNN) combines Mask and Keypoint R-CNN (He et al 2017) to simultaneously perform object detection (guidewire bounding boxes), instance segmentation (guidewire segmentations), and keypoint localization (guidewire tips).The segmentations are post-processed to estimate guidewire directions (in 2D detector coordinates).Tip and direction predictions from multiple images are then backprojected to produce candidate 3D poses, which are filtered through a correspondence process to yield final pose estimates (in 3D world coordinates).The resulting guidewire positions and trajectories can be virtually overlaid on the patient's pre-operative CT scan to provide navigation support akin to conventional navigation using surgical trackers (figure 1(d)).
Table 1 provides a summary of the mathematical notation used in this work.The sections below detail various stages of the algorithm illustrated in figure 1.

Region-based CNN for guidewire detection
A combined Mask and Keypoint R-CNN is used to simultaneously segment the guidewires and localize their tip coordinates in detector coordinates.This was accomplished by adding the Keypoint R-CNN head and predictor to a Mask R-CNN network, as shown in figure 1(b).For each detected guidewire within the predicted bounding box, the mask head produces the segmentation, while the keypoint head identifies the pixel corresponding to the guidewire tip.Non-max suppression is applied on the detected keypoints, such that for any two keypoints <6 mm apart, the one with a lower detection score is removed.
Instance segmentations and keypoints are post-processed to determine the guidewire direction in detector coordinates.Segmentations are truncated to within a fixed distance, l = 30 mm, of the detected tips to focus on the straight tip portion and reduce errors due to potential instrument deformation.This distance was obtained experimentally by varying l from 5 to 50 mm and identifying the value where the change in 2D direction error drops to <1%.The guidewire 2D direction is defined using principal component analysis of the segmented pixel coordinates, where the direction, d ,    { } for each view (i.e.rotation of the gantry) as detailed in Galigekere et al (2003).The x-ray source location is determined by decomposing the projection matrix as P P s , where [ ] * demarks matrix columns.The backprojected ray for each detected guidewire tip, i, is defined as


and is illustrated in figure 2. Each backprojected ray from the first view, j = 1, is paired with each ray from the second view, j = 2, in a combinatorial fashion.Their intersection is estimated using linear least squares, where i j , ¶ are the distances between an intersection and each ray that formed this intersection.The intersections are then considered as candidate guidewire tip locations, t , i in world coordinates (referred to as the 3D tip) and are used to establish correspondence (section 2.3).

Guidewire direction estimation
Two methods were developed, and comparatively evaluated, for estimating the 3D direction of guidewires: Cross product (CP) method is an analytical closed-form solution that computes the intersection of planes formed by the detected guidewires.A new point in the first view, p , i,1 is defined to be l = 30 mm away from the tip along the direction vector: p t ld .
Its corresponding point in the second view, p , i,2 is then identified using the epipolar constraint as described in Sprague et al (2006), Yaniv (2009).Accordingly, the location of p i,1 on the second view is at the intersection of the epipolar line, Fp , i,1 and the line formed by t d and ,  where the F is the fundamental matrix as defined in Yaniv (2009).Backprojected rays, p , i,j  are obtained as described in section 2.2.1.The method, illustrated in figure 3(a), takes the tip locations in detector coordinates, t , i j , and the associated point p i j , of each view as input.For a given view, the backprojected ray, t p and , i,1 i,1   form a plane with the normal vector n p t .
The 3D direction vector (i.e. the intersection of the two planes) is then given by the cross-product of n 1  and n : Iterative closest ray (ICR) uses a numerical optimization approach to provide added robustness to noise in 2D direction estimation and in cases where the guidewire aligns with the gantry rotation plane (comparison to CP is detailed in section 2.5.3).The ICR method, illustrated in figure 3(b), computes the ray passing through the 3D tip t i that minimizes the distance between p i,1  and p : . 3D localization of guidewire tips.Each detected guidewire, i, with tip, t , i j , on view j is backprojected to obtain t .
The optimization uses t p i i as an initial estimate.

Corresponding detections from multiple views
Correspondence of guidewires detected across different radiographic views is used to identify their tip and directions in 3D world coordinates.For each combination of backprojected rays, the distances i,1 ¶ and i,2 ¶ from the resulting intersection t i to each ray t i,1  and t i,2  that form this intersection are calculated.Intersections with sum of distances i i ,1 ,2 ¶ + ¶ greater than a threshold t are filtered, considered as unlikely to be formed by a given guidewire.To ensure that both rays are reasonably close to the intersection, the permutation of ray combinations that minimizes the sum of distances t is used, which yields the intersection t i that defines a guidewire's tip location in world coordinates.

Experimental setup, data, and network training 2.4.1. Experimental setup
The datasets used in the experiments below were created using the setup shown in figure 4(a).The Cios Spin mobile C-arm (Siemens Healthineers, Erlangen, Germany) was used to acquire CBCT images from five cadaveric specimens over three anatomical regions: chest, lumbar, and pelvis.Images were acquired at 110 kV and 1.4 mAs (mean across all acquisitions).The C-arm geometric magnification was 1.87 for an object at the isocenter (source-to-detector distance = 116.4cm and source-to-axis distance = 62.3 cm).Each scan provided 400 projections with 976 × 976 pixels and 0.304 mm isotropic spacing.CBCT images were reconstructed on a 512 × 512 × 512 voxel grid with 0.312 mm isotropic spacing.

Training dataset
As shown in figure 5, radiographic projections underlying the uninstrumented CBCT scans were used to build a dataset of 10 000 training and 1000 validation images.Images were log-transformed to stretch the histogram and invert the intensities to assign higher pixel values to the attenuating features (viz., guidewires).The dataset was augmented using random (uniform distribution) affine transformations with ±2.5°rotation and ±5 mm translation.To remove 'black pixels' introduced by the affine transform, images were scaled by a factor of 1.2 and cropped.The guidewires were simulated on real images of the anatomy, thus facilitating the creation of an arbitrarily large dataset and providing ground truth labels (segmentation and keypoints) for network training.To simulate the guidewires, random 3D cylindrical surface meshes were constructed using the B-spline model described in Uneri et al (2016).One to four guidewires were added by forward projecting the 3D models at varying locations, rotations, and deformations.Consistent with tools used in standard practice, the radius of each wire was randomized (uniform distribution) between 0.5 and 2.0 mm, the length between 200 and 300 mm, and the attenuation coefficient between 0.15 and 0.25 mm −1 , emulating stainless steel at ∼110 kV.Models of other surgical instruments commonly used in orthopaedic surgery (such as hemostats, clamps, and retractors) were added to the images with the same attenuation coefficients as the simulated guidewires (i.e.assuming steel) to create realistic radiographic scenes and increase robustness to instrument clutter.Each instrument model was first placed at the isocenter, randomly translated by ±90 mm and rotated by ±180°along each axis, and forward projected on acquired images.Although the random translations and rotations do not necessarily represent realistic positioning, the wide range of variability in size and position was expected to help generalize the trained model.Quantum noise was added in data generation, as outlined in Wang et al (2014), to emulate varying imaging protocols, simulating mA between 20% and 100% of its original value.
The R-CNN model was trained with a weighted sum of the mask and keypoint binary cross entropy loss: The weight, l = 0.1, was introduced to compensate for the difference in the magnitudes of the two losses.The batch size was set to 20, and the learning rate was set to 10 −4 .Gaussian noise was added to slightly perturb the pixel intensities of each input image, with the standard deviation chosen randomly in the interval where I min and I max are the minimum and maximum image intensity, respectively.Training was terminated when the validation F1 score stopped increasing for 5 epochs (corresponds to ∼75 epochs).

Test dataset
The test dataset comprised separate sets of projection data containing guidewires and other instruments.The images were acquired from the same five cadaveric specimens.Unlike the training and validation images that used simulated guidewires and instruments, all projections from the test dataset captured real surgical guidewires and instruments.Two guidewires were drilled into bone for each anatomical site, and a CBCT scan was acquired.The use of two guidewires was motivated by trauma surgery-a procedure that applies to all tested anatomical sites (viz., pelvis, lumbar, and chest) and involves placing guidewires on either side of a fracture site.Other surgical instruments, such as screws, retractors, and scissors, were then placed around the incision site, and a second CBCT was acquired.Compared to the training dataset where the instruments were simulated with random placement, the physical locations of these instruments were consistent with clinical practice.The presence of instrument clutter serves to both create realistic clinical scenes and helps evaluate the robustness of our algorithm to possible guidewire tip occlusions caused either by overlapping instruments or other guidewires.The test dataset was separated into two categories, each consisting of projection images underlying 18 CBCT scans: images containing exclusively two guidewires (without instrumentation clutter); and images containing guidewires along with other surgical instruments (with instrumentation clutter).

Performance evaluation 2.5.1. Guidewire detection in the presence of instrument clutter
The first experiment evaluated the accuracy of guidewire detection and its robustness to the presence of instruments other than guidewires.The ground truth guidewire tip and direction were manually delineated from CBCT reconstruction volumes and projected to 2D detector coordinates using the C-arm system geometry.This process was repeated three times for each guidewire of the dataset, and the tip and direction variability were measured to be 0.42 mm and 1.1°, respectively.Geometric accuracy was measured in 2D detector coordinates, characterized by the Euclidean distance between the ground truth (t¢) and predicted tip, Detection recall and precision were calculated and compared for the image sets with and without instrument clutter, where detections with t D <10 mm were defined as true positives.Statistical significance of the differences was tested according to the unpaired, nonparametric Mann-Whitney U test.

Accuracy of correspondence
The accuracy of the proposed correspondence method in rejecting false positive detections was evaluated in terms of correspondence recall and precision.Correspondence true positives were defined as correctly detected tips from the first view that matched the detections from the second view.Correspondence false positives were defined as any combination that included a false positive.Correspondence false negatives were defined as true positive detections removed as an outlier during correspondence.
A simulated dataset was created and used to identify the correspondence threshold, , t that maximizes the correspondence recall and precision.This dataset was created by randomly projecting 1000 guidewire models according to a circular projective geometry with a 600 mm source-to-axis distance and 1200 mm source-todetector distance (consistent with the C-arm used in experiments).Guidewires were simultaneously projected on pairs of views separated by at least 45°to create 500 image pairs, each containing two guidewires.Projected tip locations were independently perturbed in each view to emulate the variability in model prediction.Consistent with the results presented below (section 3.1), perturbations followed a bimodal distribution with a 2.1 mm mean and 1.5 mm standard deviation.Zero to six false positive tips were randomly added to each view to reproduce the detection precision of the test dataset.All tip coordinates and false positives were input to the correspondence method with t varying 1-14 mm.The identified t value was then evaluated on real images from the test dataset.

Comparison of 3D direction estimation methods
The two methods for 3D direction, CP and ICR, were evaluated with respect to the location of guidewires relative to the plane formed by rotation of the C-arm gantry.While a perfect alignment of the guidewire within the plane is not a common occurrence, a simulated dataset was created to evaluate the direction estimation accuracy with respect to the angle between the guidewire and gantry plane, referred to as 'out-of-plane angle.' Meshes of guidewires at angles from the rotation plane varying from 0°(in-plane) to 90°(out-of-plane) were simulated using a circular geometry, and forward projected.Pairs of images at various gantry angles were simulated using the same calibrations as in section 2.5.2, with two images forming a pair separated by at least 45°.The 2D tip coordinates in each view were determined using the C-arm projection matrices and the known 3D tip location.The 2D tip location of the guidewire was perturbed following the same bimodal distribution in section 2.5.2.The 3D direction was computed using the CP and ICR methods, and the median direction error and interquartile range at each out-of-plane angle were compared.

Geometric accuracy of tip and direction estimation
Geometric accuracy was measured in 3D world coordinates, analogous to its 2D counterpart in section 2.5.1.Accuracy in localizing guidewire tips and directions was assessed on radiographs with and without instrument clutter.The 3D tip error, t , W ∆ was defined as the 3D Euclidean distance between a ground truth and predicted guidewire tip, and the 3D direction error, d ,

∆
as the angle between a ground truth and predicted direction.The complete pipeline was deployed in a prototype interface for providing 3D navigation support using fluoroscopic imaging.

Guidewire detection in the presence of instrument clutter
The performance of R-CNN detection of guidewires was evaluated on the test datasets with and without instrument clutter in terms of recall, precision, and tip and direction errors.The recall for images containing exclusively guidewires was 85%, compared to 78% for images with instrument clutter (82% over both datasets).The precision was 71% and 40%, respectively (56% overall).
The median accuracy in detecting guidewire tips on the overall test dataset was 2.2 mm with interquartile range (IQR) [1.3, 3.4] mm and 1.1°IQR [0.5°, 2.2°] in directions.Figures 6(a), (b) show the distribution of tip and direction errors on projections from the test dataset, making the distinction between radiographs without and with instrument clutter.The median tip accuracies were 2.2 mm IQR [1.2, 3.3] mm and 2.3 mm IQR [1.3, 3.5] mm for radiographs without and with instrument clutter, respectively.The median 2D direction errors were 1.1°with an IQR of [0.6°, 2.0°] and 1.1°IQR [0.5°, 2.2°], respectively.The differences in the distributions were not significant (p > 0.05).
Figures 6(c), (d) illustrate example detections representative of the median tip and direction errors in detector coordinates.Figure 6(c) shows an example case in the pelvic region with no other surgical instruments present in the scene.Although this anatomical site is particularly challenging due to high-intensity cortical bones, the model achieved a tip and direction accuracy of 1.9 mm and 2.3°.Despite the presence of scissors and retractors, the accuracy for the guidewires in figure 6(d) was 2.7 mm and 3.3 mm for the tips and 0.6°and 2.8°f or the directions.

Accuracy of correspondence
Figure 7 shows the correspondence precision and recall as a function of the correspondence threshold, .
t The change in precision and recall becomes negligible for t > 4 mm on the simulation set with <1% change at a 1 mm increment of the threshold.The correspondence threshold was therefore set to 4 mm for the rest of the analyses, which yielded 84% recall and 94% precision on the simulation dataset.
Repeating the evaluation on the test dataset, the same trend was observed with respect to , t thus validating the fidelity of the simulated dataset and confirming the selected threshold.The results translated to 90% recall and 83% precision in the overall test dataset.The accuracy achieved on the subset without instrument clutter was a recall of 97% and precision of 88% compared to 81% and 77% on the subset presenting instrument clutter.

Comparison of 3D direction estimation methods
Methods for 3D direction estimation, CP and ICR, were compared with respect to the angle between the guidewires and the gantry plane (figure 8(a)).As shown in figure 8(b), the CP method achieved an average  median 3D direction error of 29°for guidewires within 10°of the imaging plane compared to 5°for the ICR, demonstrating the added robustness of the ICR method to in-plane guidewires.The results converged to similar values for larger out-of-plane angles: the CP method achieved a median error of 3.66°IQR [1.7°, 5.9°], and the ICR method a median of 3.62°IQR [2.2°, 6.7°] on the overall dataset, which contained guidewires close to the imaging plane.

Geometric accuracy of tip and direction estimation
Figures 9(a), (b) show the accuracy in estimating the guidewire tips and directions in 3D world coordinates on the test dataset.The median tip accuracy was 1.8 mm IQR [1.2, 2.9] mm.Note that these errors are lower than the corresponding errors measured on the detector (figure 6), as the objects in the detector frame are magnified according to their distance from the source.The outliers in guidewire tip estimation were caused by correspondence false positives in the presence of instrument clutter.The median direction error was 2.7°IQR [1.5°, 4.9°].The 3D direction outliers were radiographs presenting guidewires close to the C-arm gantry plane and correspondence false positives.Figure 10 illustrates an example interface for 3D navigation that would be presented to the surgeon during the intervention.Detected and localized guidewires are used to annotate the fluoroscopic images acquired as part of the standard step-and-shoot fluoroscopic workflow (figure 10(a)).Virtual guidewire models are also overlaid on the preoperative CT in figure 10(b).The CT is resliced using the estimated 3D direction to show endon and in-plane views of the guidewire.Figure 10(c) shows the same views from a postoperatively acquired CT, f (b) Guidewires were simulated for out-of-plane angles ranging from 0°( in-plane) to 90°(orthogonal to the rotation plane).confirming the provided overlay's accuracy.The 3D reckoning of guidewires from 2D fluoroscopy being very challenging and often requiring repeated acquisitions of projections, the proposed framework offers a solution to reduce radiation exposure in orthopaedic trauma surgery by providing a 3D display of a guidewire from only two radiographic projections.

Discussion
The paper presented a new method for 3D navigation of surgical guidewires that combines region-based guidewire detection from pairs of fluoroscopic images, 3D localization of guidewire tips from the intersection of backprojected rays, and a correspondence method to associate tips from different views.
The first experiment consisted of evaluating the robustness of the R-CNN guidewire detection to instrument clutter.The proposed combined Mask and Keypoint R-CNN model detected guidewires with a precision of 71% for images without instrument clutter and 40% on projections that presented instrument clutter.While the presence of instrument clutter introduced false positive detections and decreased the model precision, most of these false positives were filtered out by the proposed tip correspondence method.Indeed, an experiment was conducted to evaluate the robustness of the proposed tip correspondence method to false detections and yielded a precision of 83% on the overall test dataset as opposed to before correspondence.The remaining correspondence false positives were challenging scenes where two tips were very close to each other or located on the same plane.The correspondence recall obtained was 90%, which means that some of the true positive detections were filtered out as outliers by the correspondence threshold.Esfandiari et al (2018) proposed an alternative view correspondence method based on epipolar lines.This method requires estimation of the C-arm calibrations using a phantom and 3D-2D registration.The proposed approach takes advantage of CBCTcapable motorized (and, more importantly, encoded) C-arms that permit the use of prior system calibrations.While this limits the application to modern C-arm designs, offloading the computationally costly calibration process can help provide rapid pose updates.Compared to von Atzigen et al (2021), the proposed correspondence method makes no assumptions about the C-arm gantry or its trajectory.The triangulation is expected to work as long as view angles are separated by at least 10°-15° (Uneri et al 2014).
The proposed 3D direction methods were tested for their ability to handle in-plane guidewires.Results showed that the ICR method had a lower 3D direction error than the CP method, with median direction errors of 11°and 60°, respectively.This difference can be explained by the fact that the CP method provides a closed solution to the intersection of planes) while ICR iterates to find the minimum distance between backprojected rays.While ICR is more robust to lower out-of-plane angles, the dependence on the estimated tip location makes it susceptible to tip errors.For optimal performance, both methods could be used together such that the crossproduct is used nominally unless a low out-of-plane angle is detected, in which case ICR is invoked.Future work will consider a 3D network architecture with a 3D regression block such that a guidewire's tip coordinates and 3D direction can be directly computed as part of the network architecture.
Overall, the proposed method achieved a median tip accuracy of 1.8 mm IQR [1.2, 2.9] mm and direction accuracy of 2.7°IQR [1.5°, 4.9°] in 3D world coordinates.The method is intended for step-and-shoot fluoroscopy, which is consistent with the current workflow for orthopaedic trauma surgery.Accordingly, the approach operates in near-real-time (i.e. as new image pairs are acquired) with an overall algorithm runtime of 0.73 ± 0.11 s.The guidewire trajectories can be visualized as an overlay on a patient's preoperative CT scan.This overlay allows the surgeon to visualize the position of a guidewire with respect to the patient's anatomy by acquiring two radiographs rather than a CBCT.Since the algorithm is agnostic to views-provided they are sufficiently separated-, repeat imaging for locating specific anatomical views (commonly referred to as 'fluoro-hunting') may be avoided, which can result in further dose reduction.These observations warrant and motivate a separate study to evaluate the potential reduction in fluoroscopic exposure time to the patient and the surgical staff.
Although the study was evaluated using only two implanted guidewires, it should be noted that tip and direction estimations are independently performed for each detection with no assumption on the number of guidewires and no anticipated impact on accuracy.A greater number of guidewires, however, would increase the chances of occlusion and impact robustness.In our study, this scenario was accounted for by the addition of instrument clutter that creates such occlusions.The amount of instrument clutter may differ from clinical cases the projections of our test dataset contained at most three instruments other than guidewires.Future work will therefore include clinical data for training and testing.Future work would also extend the number of anatomies present in the training and testing datasets to account for the variability in the use of guidewires.

Conclusions
A new method for 3D navigation of surgical guidewires from fluoroscopic images was developed and evaluated in preclinical studies.The proposed solution uses an R-CNN for detecting guidewire tips and directions, and a model-based approach for 3D localization and correspondence.The detection method achieved a median tip accuracy of 2.2 mm and median direction accuracy of 1.1°(in 2D detector coordinates).Correspondence of guidewires from multiple views demonstrated robustness to false positive detections, yielding a precision of 83% and recall of 90%.Compared to the CP method, the ICR method demonstrated superior robustness to in-plane guidewires, achieving an average median 3D direction error of 5°, while the CP method achieved an average median 3D direction error of 29°for guidewires within 10°of the imaging plane.Guidewires were localized with a median tip accuracy of 1.8 mm and direction error of 2.7°(in 3D world coordinates).The resulting guidewire location and direction can be visualized as an overlay on a patient's preoperative CT scan during the intervention.This overlay would allow the surgeon to visualize the position of a guidewire with respect to the patient's anatomy by acquiring two radiographs rather than a CBCT, thus decreasing the fluoroscopic exposure times.

Figure 1 .
Figure 1.Algorithm for detecting, localizing, and corresponding guidewires from fluoroscopic imaging.(a) Fluoroscopy images acquired at two view angles are taken as input.(b) An R-CNN detects guidewire tips and directions in detector coordinates.(c) Localization and correspondence of guidewire tips in world coordinates is established by backprojection of the tips and directions.(d) The resulting pose estimates are visualized in a navigation software as overlays on preoperative (or intraoperative, preinstrumentation) 3D images-e.g.CT or CBCT.

2. 2 .
Model-based guidewire localization 2.2.1.Guidewire tip localization using backprojection The method for 3D guidewire localization uses the geometric model of the imaging system.Such models are commonly available in C-arms used for 3D CBCT imaging and are obtained through a geometric calibration process (Cho et al 2005) ahead of time.The geometry is represented as a set of projection matrices, P , q pair of rays is computed via least-squares minimization to define candidate guidewire tip locations, t , i in 3D world coordinates.

Figure 3 .
Figure 3. Estimation of 3D direction using the (a) cross product (CP) and (b) the iterative closest ray (ICR) methods.The CP method has the advantage of geometric simplicity and speed; the ICR method was hypothesized to be more robust for cases in which the guidewire was nearly coplanar with the C-arm rotation plane (for which the cross-product of backprojected rays is zero).

Figure 4 .
Figure 4. Experimental setup and example fluoroscopy image pairs.(a) Images were acquired using 5 cadaver specimens with a mobile C-arm Cios Spin.An anthropomorphic phantom is shown in (a) for illustration).(b) Example images with guidewires implanted in the chest, lumbar, and pelvis regions.(c) Projection data were acquired with and without the placement of additional surgical instruments (e.g.scissors and retractors) to evaluate the robustness of the approach to radiographic 'clutter'-i.e.instruments that were neither anatomy nor guidewires.

Figure 5 .
Figure 5. Detailled illustration of the data generation and augmentation pipeline for fluoroscopic projections of the (a) chest, (b) lumbar, and (c) pelvic regions.Uninstrumented projections (without guidewires or other instruments) were used as input and underwent a series of random transforms, including log conversion, cropping, rotation, translation, and scaling.Guidewire and surgical instruments were simulated by forward projecting their 3D digital models onto the 2D images according to the calibrated system projection geometry.

Figure 6 .
Figure 6.Geometric accuracy of guidewire tip and direction estimation in 2D detector coordinates.(a) Tip and (b) direction errors comparing the accuracy and precision in images without and with instrument clutter (c)-(d).Examples illustrating the median performance (c) in the pelvic region without instrument clutter and (d) in the lumbar region with instrument clutter.

Figure 7 .
Figure 7. Correspondence recall and precision for thresholds ranging 1-14 mm.Results on (a) simulated detections, (b) the test dataset, (c) test images without instrument clutter, and (d) test images with instrument clutter.The nominal value of the correspondence threshold (t 4 mm) is marked with a solid black circle.

Figure 8 .
Figure 8. Robustness of 3D direction estimation for guidewires that are nearly coplanar with the C-arm plane of rotation.(a) Illustration of guidewires placed out-of-plane by an angle .f (b) Guidewires were simulated for out-of-plane angles ranging from 0°( in-plane) to 90°(orthogonal to the rotation plane).

Figure 9 .
Figure 9. Geometric accuracy in localizing the guidewire (a) tips and (b) directions in 3D world coordinates.Boxplots indicate the median, IQR, and range, with a violin plot superimposed to illustrate the full extent of the underlying measurements.The median tip accuracy was 1.8 mm IQR [1.2, 2.9] mm and the median direction error was 2.7°IQR [1.5°, 4.9°].

Figure 10 .
Figure 10.Example interface for 3D navigation from a pair of intraoperative fluoroscopic images.(a) The intraoperative radiographs are taken as input to the proposed detection and localization methods.(b) Surgical guidance is provided by rendering the computed tip and direction of the guidewire in 3D world coordinates within preoperative CT or CBCT.(c) Postoperative CBCT (with the guidewire in place) illustrates the accuracy of intraoperative overlay.The example shown corresponds to the median performance shown in figure 9.

Table 1 .
Summary of the mathematical notation used in this work.Bold symbols denote 3D coordinates.