Tissue texture extraction in indocyanine green fluorescence imaging for breast-conserving surgery

Breast cancer is the most prevalent cancer among the female population worldwide [1]. The preferred surgical intervention in the majority of women with small unifocal tumours is breast-conserving surgery (BCS), which when combined with radiotherapy provides equivalent cancer treatment outcomes when compared to mastectomy (removal of the entire breast) [1]. Compared to mastectomy, BCS is a less traumatic surgery, associated with fewer complications, quicker recovery time, as well as improved cosmetic outcome [2].

BCS entails resecting the tumour en bloc, inclusive of a layer of normal tissue (a.k.a. 'the margin'). However, if the post-operational histopathology reveals positive resection margins, which occurs if the excised specimen's margins contain malignancy, re-operation may be indicated. Cancer recurrence has been shown to correlate with positive margins, as without further surgery to resect the remaining tumour burden, local control of the disease is not achieved [3]. However, reoperation comes with a cost for both the patient and the healthcare system. It has been associated with additional physiological and psychological distress, poor cosmesis, risks post-operative complications (e.g. surgical site infection) and the need for increased human and financial resources [4]. Therefore, it is crucial to improve the intraoperative assessment of resection margins to better guide the index surgery towards decreasing re-operation rates, which are reported to be as high as 27% in the UK [5] and 21.6% in the USA [6].

Despite the plethora of available methods for tumour guidance and intraoperative margin assessment, re-operation remains an unresolved challenge [7]. To date, there is no clinically accepted technology capable of providing the surgeon with real-time intraoperative visual feedback on tumour location, size, and invasiveness. However, fluorescence guided surgery is a promising contender in tackling the existing challenge of high BCS re-operation rates. Fluorescence imaging involves the use of a clinically approved contrast agent (fluorophore), often given via injection. Most of the available agents are considered safe and have minimal side effects [8]. Upon excitation of this agent with non-ionising illumination, the fluorescence is captured with the use of ergonomic and user-friendly camera equipment [9]. The fluorescent signal is then processed and displayed so that the surgeon can infer whether further resection is required and if so, where. Fluorescence guided surgery is capable of yielding good cancer sensitivity scores [10] and has the potential for quantitative outcomes [11].

1.2.1. Previous work and contribution.

Both the hardware and software have been iteratively developed in collaboration with the clinical team. The hardware encompasses a filtered Xenon light source (Asahi Spectra Max-303, Asahi SpectraCo., Ltd, Tokyo, Japan) for surgical field illumination via a flexible fibre optic illuminating ring (Mini Annular Ringlight MA3172, Vision Light Tech B.V., Protonenlaan, Netherlands) and a two-camera setup that captures colour (FLIR Grasshoper3 GS3-U3-28S5C-C, FLIR Systems, Inc., Wilsonville, OR, USA) and monochrome (FLIR Grasshoper3 GS3-U3-28S5M-C, FLIR Systems, Inc., Wilsonville, OR, USA) images simultaneously. Both cameras have 2/3'' CMOS sensors with a resolution of 1920 × 1440 pixels. The light path was divided into two branches corresponding to the two cameras, using a 50:50 beam-splitter (CCM1-BS013/M, Thorlabs, Inc., Newton, NJ, USA). For ICG fluorescence excitation, a bandpass filter 500–750 nm (combination of 500 nm long-pass and 750 nm short-pass) was used (TECHSPEC OD4, Edmund Optics, Inc., Barrington, NJ USA). For the fluorescence emission, a bandpass filter (825 nm central wavelength, 50 nm FWHM, Edmund Optics, Inc., Barrington NJ, USA) was installed in front of the monochrome camera. A 35 mm SLR camera photographic lens with variable focal length 28–70 mm (Sigma Co, Japan) was used for focusing and imaging. The user interface for setting up an acquisition and visualisation was developed in LabVIEW™ software. A schematic diagram explaining principal functions of the device and its use in the operating theatre for the experiment is shown in figure 1.

Zoom In Zoom Out Reset image size

The system described above has been tested during BCS in a 6 month clinical feasibility trial (research ethics committee, REC 18/LO/2018). Overall, ten patients provided informed consent and were recruited with a mean age of 56 (range 15–70) years, and a mean body mass index of 24.2 (range 19.2–30.2). Eight of the patients had invasive ductal carcinoma, and of those, five had concomitant ductal carcinoma in situ, one had invasive lobular carcinoma with concomitant DCIS, and one had invasive mucinous carcinoma. Receptor status was 90% ER-positive, 70% PR-positive and 10% Her2-positive.

Patients were injected with 12.5 mg of ICG (Verdye, Diagnostic Green GmbH, Otto-Hahn-Str. 20, 85609 Aschheim-Dornach Germany) intravenously, once the skin flaps had been raised to expose the tumour. The excised tumour specimens were imaged from six projections: anterior, posterior, superior, inferior, medial and lateral. At least three images for each projection were acquired to decrease the probability of blurring due to motion. The sample surface was perpendicular to the optical axis to reduce the image distortion due to sample surface curvature. Image sequences were acquired of the tumour in situ, the resected tumour, any resected SLNs, and the surgical cavity post-resection. The excised tumour specimen was also imaged using the gold standard 'FaxitronTM OR Specimen Radiography' system (Hologic, Marlborough, USA) for intraoperative evaluation by the operating surgeon. These high-resolution images using ionising radiation (similar to preoperative mammography) enable rapid assessment of tissue density, as cancer and precancerous conditions are more dense than normal breast tissue. These images are formally reported by radiologists postoperatively. All specimens were sent for formal histopathology for the final identification of specimen margin status. Histopathology results served as the gold standard for the evaluation of the classification models used. A summary of the workflow is presented as a schematic overview in figure 2.

Zoom In Zoom Out Reset image size

ICG emission and absorption spectra, together with breast autofluorescence spectra, were needed to optimise the imaging setup, especially the combination of fluorescence excitation and emission filters. Spectral acquisition of fluorescence emission and excitation spectra in bovine plasma was obtained via a laboratory spectrofluorometer (Fluorolog-3, model FL-21 HORIBA Instruments Inc., Edison, NJ USA). ICG was prepared by dilution in distilled water according to the manufacturer's instructions. It was then diluted in reconstituted bovine plasma (P4639, Sigma-Aldrich Co. LLC, USA) down to a concentration of 2.5 mg ${{\text{l}}^{ - 1}}$ or 3.23 × ${10^{ - 6}}{ }$ M. This is the expected in vivo ICG concentration in blood, assuming that patients are injected with 12.5 mg of ICG, that the total blood volume for an average adult is 5 l and that the ICG molar mass is 774.96 g ${\text{mo}}{{\text{l}}^{ - 1}}$. We recorded fluorescence excitation spectra in the range of 400–820 nm of three samples of ICG in bovine plasma, with the emission maximum at 840 nm.

ICG fluorescence spectra in breast samples ex vivo were recorded using a compact spectrometer (Flame, Ocean Optics Inc. Rochester, NY, USA) with fitted 800–850 nm bandpass filter (Edmund Optics, Inc., Barrington NJ, USA). The illumination source was identical to that for intraoperative imaging. Thirty spectra of connective tissues, fat and tumour from a freshly excised specimen from one patient were collected after angiographic phase ICG administration at a dose of 0.25 (mg kg⁻¹). An example is presented in figure 4(b) of section 3.1.

Breast tissue autofluorescence was recorded using a compact spectrometer (USB4000, Ocean Optics Inc.) with a 425 nm long-pass filter installed (Edmund Optics). For excitation we applied a home-built setup with 405 nm laser diode (DL5146-101 S, Thorlabs Inc., NJ, USA) and autofluorescence was collected using a bifurcated Y-shaped fibre optic probe (LEONI Fiber Optics GmbH, Germany). Twenty spectra were collected from connective tissues, fat and tumour of freshly excised breast cancer specimen and lymph nodes. Normalised typical spectra, collected from three separate macroscopically normal sites of the specimen, are presented in figure 4(b).

The raw fluorescence images were processed using image texture metrics as predictors for the classification models to allow a comprehensive investigation of tumour classification accuracy. The results from this image texture-based classification were evaluated for their accuracy with the use of the receiver operating characteristic (ROC) analysis. Raw fluorescence signal intensity overlays on the colour images of the surgical scene were also used for qualitative comparison with the image texture classification results and ground truth data. All image processing was implemented in Matlab software (Mathworks, Inc., Massachusetts, USA) and a summary of the image processing steps is presented in figure 2.

2.4.1. Extraction of ground truth.

To train and evaluate the classification models and overlays, ground truth data indicating tumour and healthy tissue locations were required. This ground truth was derived from the histopathology results after identification of the specimen locations in the fluorescence images. This procedure was aided by the comparison between the specimen fluorescence images and the intraoperative specimen radiography images, which provided an anterior–posterior view of the specimen in its entirety. In the example in figure 3(A), the tip of the wire indicates the location of the tumour core while the double and triple staples indicate the superior and lateral aspects of the specimen, respectively. The corresponding histopathology results for this case determined that the tumour was 1 mm from the posterior margin, 3 mm from the anterior margin and more than 5 mm from the remaining superior, inferior, medial and lateral margins. Therefore, the tumour location (red contour, figure 3(D)), as well as the healthy tissue (green marks, figure 3(D)), were only marked in the fluorescence and colour images of the posterior side of the specimen—as the tumour was considered too deep (3 mm) on the anterior side. For the same reason cases 3–6 and 10 were excluded from this analysis since the cancer was too deep for NIR detection.

Zoom In Zoom Out Reset image size

2.4.2. Texture metrics.

We recorded fluorescence excitation spectra of three ICG samples in bovine plasma with a concentration 2.5 mg ${{\text{l}}^{ - 1}}$ (3.2 × ${10^{ - 6}}{ }$ M). A normalised and filtered excitation spectrum is presented in figure 4(A) (spectral intensity variance was of the order of several per cent, accepted as negligible, not shown) alongside a normalised absorption spectrum of ICG in plasma in concentration 6.5 × ${10^{ - 6}}{ }$ M (5 mg ${{\text{l}}^{ - 1}}$) adapted from Landsman et al [18]. Figure 4(B) presents a typical normalised ICG spectrum in plasma in vitro and an example ICG spectrum from a freshly excised breast specimen ex vivo showing a maximum at 814 nm. Certain ICG spectra from fat and connective tissues demonstrated variability of the maximum position in the range 815–820 nm (not shown for clarity). Breast autofluorescence spectra are also presented for 405 nm excitation.

Zoom In Zoom Out Reset image size

The NIR system resolution was estimated using a USAF 1951 resolution test target (Thorlabs Inc. Newton, NJ, USA). With a field of view of 6.3 × 8.5 cm at 40 cm working distance the device resolution is seven line pairs per mm. Thus, the device is potentially capable of recording features at sub-mm scale as demonstrated in figure 5.

Zoom In Zoom Out Reset image size

The in vitro system sensitivity has previously been shown to reach concentrations down to the nanomolar scale using aqueous ICG solutions. The intraoperative raw fluorescence images of the tumour in vivo further show that the system is sufficiently sensitive for in vivo ICG detection, as shown in figure 6 and supplementary video 1 (available online at stacks.iop.org/JPD/54/194005/mmedia). Specifically, at a working distance of 10 cm (the minimal acceptable safe distance in the operating theatre), an in vitro concentration of 40 × ${10^{ - 9}}{ }$ M (0.14 mg ${{\text{l}}^{ - 1}}$), was the lowest detected, considering as a threshold a signal to background ratio of 3.28 [19]. Thus this setup was able to detect ICG down to a concentration that is approximately 20 times lower than that expected in blood. Apart from the images of the tumour in vivo in figure 6, overlaid images of the fluorescence ex vivo are presented in figure 7. Both of the figures show the overlay of the green colour map extracted from the raw fluorescence image pixel values for qualitative comparison with the image texture classification results and cancer ground truth as described in section 2.4.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The results of Whitney U tests upon 'case-wise' analysis are highlighted in table 2 for the cases where extraction of cancer ground truth was feasible.

p-Values showed a similar trend irrespective of the side length of the sampled squares and only the 64-pixel results are presented in the main paper with full results in table 1S of the supplementary material. In most of the cases, the slope and intercept of linear fit of PSD curves were significantly different between cancer and healthy tissue. When p < 0.05 (in bold) we conclude that there is a significant difference between cancer and healthy samples.

Table 3 presents the ROC analysis results for the different combinations of classification models and predictors. The results were found to be independent of the size of the square-shaped patch and only the results for the side length of 64 pixels are presented in this section since this outperformed the others in most cases. Moreover, both the 'case-wise' and the 'cross-validation' approach are depicted for comparison. As an example of the ROC data, the curve for the 9th case is presented in figure 8 for the logistic regression model and 'case-wise' analysis.

Zoom In Zoom Out Reset image size

The prediction scores calculated with the use of the trained models described in section 2.4.2 are displayed as green overlay maps on the corresponding colour image of the field of view in figure 9. The example is case ID: 09 to match figure 7.

Zoom In Zoom Out Reset image size

There is good similarity between the excitation and normalised absorption spectral shapes in figure 4(A). The excitation maximum is at 806 nm, while the absorption peak is shifted toward a slightly longer wavelength of 810 nm. Some spectral shape differences are observed between 700 and 750 nm and the fluorescence excitation efficiency is lower for this region than the absorption spectra, but the difference is not dramatic. This similarity between absorption and excitation spectra justifies the use of the band 700–750 nm for ICG fluorescence excitation in plasma.

Regarding figure 4(B), the maximum of ICG emission in plasma excited at 720 nm is observed at 804 nm. It is shifted toward shorter wavelength compared to ICG emission maximum in plasma in vitro (820 nm), reported by Ruiz et al [21]. One possible reason for this difference could be the ICG concentration. The observed ICG emission maximum in vivo correlates with results reported by Haj-Hosseini et al [22] for freshly excised lymph nodes, namely maxima were in the range around 820 nm, although in some cases the maximum was in the range 850–860 nm. The observed emission maxima shift, perhaps, could be explained by ICG redistribution between serum proteins and the surface components or within the vascular endothelium [23].

To estimate how tissue autofluorescence could potentially affect ICG fluorescence we performed autofluorescence measurements at 405 nm excitation. Tissue autofluorescence could affect the contrast in the fluorescence images. This suggested that the autofluorescence excited by the short wavelength end of the Xe lamp would mostly be observed in the range 425–750 nm, but in the case of weak ICG fluorescence signal the autofluorescence could potentially interfere in the 800–850 nm range. To avoid such artefacts, we limited the excitation/visualisation broadband light bandpass range to 500–750 nm.

As demonstrated in section 3.2, the system can detect ICG in vitro down to an aqueous concentration in the nanomolar range as well as in vivo during BCS. Patients were injected with 12.5 mg of ICG which corresponds to an ICG concentration in blood of 2.5 mg ${{\text{l}}^{ - 1}}$ (3.2 × ${10^{ - 6}}{ }$ M), however, the qualitative review of the fluorescence images has shown that the fluorescence signal does not always correlate with the location of the tumour. This can be observed by comparing the ground truth cancerous tissue marked in figure 3 with the simple overlays of the fluorescence signal in figure 7. One possible explanation for this is that the in vivo and ex vivo images of the tumour were acquired immediately after the ICG injection during the angiography phase, thus not providing sufficient time for any EPR to take effect. The timing aspect of the imaging protocol requires further investigation.

Instead, in this work, we assume that the fluorescence signal may indicate the local vasculature and not necessarily the tumour. ROC analysis will be used to investigate the EPR effect using immediate and delayed ICG injection during an ongoing clinical trial of the system (REC 19/LO/0927).

There is a significant difference in the slope and intercept of the linear fit of the PSD curves between the cancerous and normal tissue, as shown in table 2. This is not the case for the Euler or fractal dimension values (p > 0.05, with the exception of case ID: 01) in contrast with [16]. However, p-values here are extracted on a patient-wise approach whereas in [16] inter-patient bias has not been investigated due to small sample size. The above behaviour is observed independently of the dimensions of the square-shaped patch used (supplementary table 1S). In agreement with the above, the PSD-derived slope and intercept outperform the Euler and fractal dimension metrics as classification model predictors, as shown in table 3. Furthermore, the logistic regression can achieve a better classification accuracy than the SVM model across all predictors and cases, as well as for both 'case-wise' and 'cross-validation' analysis (table 3). In the optimal operating point of the logistic regression model ROC curves both sensitivity and specificity reached values as high as 100%. Recent FGS work in BCS capitalising on the ICG EPR effect has reported a sensitivity that could also reach up to 100% [14]. However, there were some false positives and lower specificity, with the highest value of 73.9% having been reported by Pop et al [15]. AUC results of 'cross-validation' may indicate that this analysis is of low classification value. However, no conclusion can be made at this point since more data are needed to train the model to account for inter-patient variation (samples from only five patients have been used here).

The classification models do not produce calibrated probability scores [20], thus a classification threshold of 0.5 may not be the optimal choice for classifying the data used in the ROC analysis. The optimal classification thresholds (table 3) were chosen by equalising the cost for false positives with the cost for false negatives and only accounting for the relative number of cancer patches in our ground truth dataset [20]. Therefore, the suggested thresholds vary from more preservative (e.g. 0.72 for case 9, logistic regression model and 'case-wise' analysis) to more liberal (e.g. 0.47 for case 2, logistic regression model and 'case-wise' analysis).

When the logistic regression model was used in combination with the PSD-derived slope and intercept model predictors and with the 'case-wise' analysis (figure 9(B)), the indication for the location of the tumour coincides with the ground truth, indicated with the red circle in figure 3(D). On the other hand, the rest of the combinations fail to yield a meaningful overlay correspondence as confirmed by the AUC results. For example, in case ID: 09 an AUC < 0.7 was observed for the 'cross-validation' analysis which justifies the absence of correspondence for the overlays. However, although the logistic regression model in combination with the image primitives predictors and the 'case-wise' analysis yields a good AUC score, the corresponding image (figure 9(A)) has no meaningful overlay correspondence. One possible explanation for this disagreement could be that the ROC analysis has been implemented only with the use of the ground truth data, whereas for the overlay creation, the model has been used for predictions over the entire image.

In this feasibility study 10 patients were recruited overall and more data is required to improve the statistical reliability and the training of the classification models. However, the statistical analysis can be used to calculate the required sample size of an in vivo clinical validation study (REC 19/LO/0927) to achieve a study power (probability to correctly reject the null hypothesis) of 90%. Moreover, the use of ICG NIR imaging for tumour demarcation to obtain information of tissues located under superficial layers is limited. In our case, the ideal depth of interrogation to visualize tissue structure is up to several mm. As Ruiz et al [21] demonstrated on tissue mimicking phantoms, ICG fluorescence could be detected from a depth up to 3 mm. Therefore, ex vivo samples with a tumour deeper than 3 mm could not be considered in our analysis. Surface curvature (resulting in variation in surface normal or working distance) may affect and distort the images due to length scale variations from surface orientation, limited depth of field and image aberrations. To minimise these effects, we oriented the specimens such that the surface was as flat as possible. The images acquired from the six projections partially overlapped while curvature distortions were kept low. Finally, imaging of the histopathology cut up instead of the whole excised specimen would allow for more reliable ground truth extraction. In this work, the intraoperative specimen radiography images have served as the connection between the histopathology results and the fluorescence images.