A Machine-learning Method for Identifying Multiwavelength Counterparts of Submillimeter Galaxies: Training and Testing Using AS2UDS and ALESS

The UKIDSS-UDS field (R.A./decl.: 02h, −05°; Figure 1) was mapped with the SCUBA-2 bolometer camera (Holland et al. 2013) on the JCMT at 850 μm as part of the S2CLS. We provide a brief overview here; the full details of observations, data reduction, and catalog are described in Geach et al. (2017). The coverage of 0.96 deg² in UDS is relatively uniform, with instrumental noise varying by only ~5% across the field (Figure 3 in Geach et al. 2017). The final matched-filtered map has a noise of ≤1.3 mJy beam⁻¹ rms over 0.96 deg² and 0.82 mJy beam⁻¹ in the deepest part. The empirical point-spread function (PSF) has an FWHM of 148. Geach et al. (2017) identified a total of 716 submillimeter sources above a 4σ limit with a false-detection rate of ~2% (Figure 13 in Geach et al. 2017).

Zoom In Zoom Out Reset image size

We also employ a second single-dish survey sample in our analysis as an additional test of our method. This sample comprises the 126 submillimeter sources with a single-to-noise ratio (S/N) > 3.7 from the LABOCA ECDFS submillimeter Survey (LESS; Weiß et al. 2009) taken with the Atacama Pathfinder Experiment (APEX) telescope. This 870 μm map covers 0.25 deg² with a 192 FWHM and a 1σ depth of S_870μm = 1.2 mJy. The properties of this sample are thus similar to those of the S2CLS UDS sample but in a completely independent field with different multiwavelength coverage and photometric selection. We refer the reader to Weiß et al. (2009) for the details of these observations.

Band 7 (870 μm) observations have been obtained with ALMA of all 716 submillimeter sources from the S2CLS UDS map, which are described in full in S. M. Stach et al. (2018b, in preparation). Observations of 30 of the brightest (S_850μm ≥ 8 mJy) single-dish sources were undertaken in Cycle 1 as part of a pilot project, 2012.1.00090.S (Simpson et al. 2015a, 2015b, 2017), while observations of the bulk of the sample were obtained through the Cycle 3 project 2015.1.01528.S and the Cycle 4 project 2016.1.00434.S. The Cycle 1 pilot observations relied on an early interim map, and in the 30 ALMA maps, 52 SMGs were detected at ≥4σ significance (Simpson et al. 2015a, 2015b). However, in the final SCUBA-2 maps, three of these 30 sources fall below our sample selection criteria, leaving 27 of them in our final sample of single-dish-detected submillimeter sources. In Cycles 3 and 4, we observed the remaining 686 sources with S_850μm ≥ 3.5 mJy from the final S2CLS map (S. M. Stach et al. 2018b, in preparation). These observations achieve typical 1σ depths of σ_870μm ~ 0.25 mJy with synthesized beams of 015–03. The ALMA maps are tapered to ~05 resolution before sources are identified. Across all 716 single-dish submillimeter sources, we detect 695 SMGs above >4.3σ (corresponding to a false-detection rate of 2%). We refer to our complete 870 μm ALMA survey of 716 SCUBA-2 sources in the UDS field as the "AS2UDS" survey. We note that the ALMA primary beam of our observation is 174, which encompasses the area of the SCUBA-2 beam. Full details of the observation, data reduction, source detection, and cataloging are presented in S. M. Stach et al. (2018b, in preparation).

Among the 716 ALMA maps, 108 do not contain any ALMA-identified SMGs at >4.3σ. We label these as "blank-ALMA" maps. In the remaining 608 ALMA maps, we detected 695 SMGs with fluxes from S_850μm = 0.89 to 30 mJy. In the following, these maps are described as "maps with ALMA ID."

The goal of this study is to develop a method to reliably and robustly identify counterparts to single-dish-detected submillimeter sources in wide-field surveys by utilizing the multiwavelength properties of the sample of ALMA-identified SMGs. Therefore, we include the multiwavelength galaxies lying within the 108 "blank-ALMA" maps in our analysis to guarantee the completeness of our parent single-dish sample.

In our analysis, we will use independent subsets of the AS2UDS SMG sample to test the reliability of our method. We also include an additional sample for this purpose: the ALMA follow-up of the LESS survey. The ALESS survey obtained ALMA 870 μm observations in Cycle 0 of 122 of the 126 LESS sources (Hodge et al. 2013). These early ALMA observations have a typical synthesized beam of ~16 and 1σ depths of ~0.4 mJy but with a wider range of data quality than the later AS2UDS survey. For this reason, in this work, we only use the 88 "good-quality" ALMA maps from Hodge et al. (2013) to construct our test sample. Again, these include 19 "blank-ALMA" maps, which lack detected SMGs. These 88 maps yield a sample of 96 ALMA-detected SMGs with multiwavelength coverage from Simpson et al. (2014), which we will employ in our analysis. We note that the properties of this test sample differ from those of the AS2UDS sample, as it is based on an IRAC-selected photometric catalog, as opposed to the K band for AS2UDS, and the photometric redshifts are derived using different codes in the two fields. This comparison is intended to illustrate the results that would be obtained if a training set from one field were simply applied directly to a sample selected from a second survey, with different selection and photometric coverage.

Since radio synchrotron emission arises from supernova remnants, it provides a powerful tracer of obscured star formation. As such, radio emission has been traditionally used to identify counterparts to SMGs (e.g., Ivison et al. 1998, 2002).

In this work, we exploit the VLA observations of the UDS at 1.4 GHz (21 cm), which were carried out by the UDS20 survey (V. Arumugam et al. 2018, in preparation). These VLA observations cover an area of 1.3 deg² centered on the UDS field. The typical rms noise across the full VLA map is 10 μJy, and it is 7 μJy beam⁻¹ at its deepest point in the center. In total, 6861 radio sources are detected above 4σ. The details of the observations, data reduction, and catalog will be discussed in V. Arumugam et al. (2018, in preparation). In total, 714/716 ALMA pointings fall within the VLA map (Figure 1).

Deep near-infrared imaging data are crucial for investigating the properties of SMGs because of their high redshifts and dusty nature. The UKIDSS-UDS represents one of the deepest near-infrared imaging surveys over a wide area, covering 0.8 deg². As shown in Figure 1, ~90% (643/716) of our ALMA pointings are covered by UKIDSS.

The near-infrared image we exploit in our analysis is taken from UDS Data Release 11 (DR11; O. Almaini et al. 2018, in preparation), which represents the final UDS release over the whole field. Details of observations, data reduction, and catalog extraction will be presented in the forthcoming UDS data paper (O. Almaini et al. 2018, in preparation). Briefly, the DR11 reaches 3σ median depths of J = 26.2, H = 25.7, and K = 25.9 mag, which are measured in a 2'' diameter aperture. In total, 296,007 sources were detected from the K-band image using SExtractor (Bertin & Arnouts 1996), with the photometry in the J and H bands obtained in SExtractor dual-image mode.

The Y-band data are from the Visible and Infrared Survey Telescope for Astronomy (VISTA) Deep Extragalactic Observations (VIDEO) survey with 3σ depths of Y = 25.3 mag (Jarvis et al. 2013). The optical B-, V-, R_c-, i'-, and z'-band observations of UDS were carried out using the Suprime-Cam on the Subaru telescope (Furusawa et al. 2008) with 3σ depths of B = 28.4, V = 27.8, R_c = 27.7, i' = 27.7, and z' = 26.6 mag in 2'' diameter apertures. The field was also observed by the Megacam on the Canada–France–Hawaii Telescope (CFHT) in the u' band to a 3σ limiting depth of u' = 27.3 mag, again in a 2'' diameter aperture.

The mid-infrared observations of the UDS were taken with IRAC and at 24 μm with MIPS by the Spitzer Legacy Program (SpUDS; PI: J. Dunlop). The 5σ depths of the IRAC 3.6 and 4.5 μm observations are [3.6] = 24.2 and [4.5] = 24.0 mag.

In total, 12-band data (UBVRIzYJHK[3.6][4.5]) are utilized to derive photometric redshifts for the 296,007 K-band-detected sources. Details of the photometric-matched catalog and color measurement will be described in W. Hartley et al. (2018, in preparation). Hartley et al. used Easy and Accurate Redshifts from Yale (EAZY; Brammer et al. 2008) to estimate the photometric redshift for the K-band-detected sample. To obtain unbiased and high-quality photometric redshifts, they only considered those sources within the joint IRAC (SpUDS) and UKIDSS coverage and excluded those sources that have contaminated photometry (i.e., due to halos from bright stars or other artifacts). In total, ~85% (607/716) of the ALMA pointings fall in the region for which reliable photometric redshifts are available. Photometric redshifts were derived in the manner described by Simpson et al. (2013; see also Hartley et al. 2013; Mortlock et al. 2013). Hartley et al. compared the estimated photometric redshifts of ~6500 sources with available spectroscopic redshifts in the DR11 and found that the accuracy of the photometric redshifts is $| {z}_{\mathrm{spec}}-{z}_{\mathrm{phot}}| /(1+{z}_{\mathrm{spec}})$ = 0.019 ± 0.001.

The radio, optical, and near-infrared observations of our independent test sample in the ECDFS are presented in Simpson et al. (2014). The VLA 1.4 GHz data used in Simpson et al. (2014) and this work are from Miller et al. (2008). We use the radio catalog from Miller et al. (2008) to identify radio counterparts to IRAC-based galaxies in the ECDFS. Biggs et al. (2011) rereduced the VLA 1.4 GHz imaging data in the ECDFS and created a deep radio catalog containing sources down to an S/N of 3 for searching radio counterparts to single-dish-detected SMGs. We also use this deep radio catalog in our analysis to calculate the completeness of radio identification in the ECDFS. The depth and quality of the multiwavelength coverage of the ECDFS is broadly comparable to that available for the UDS in terms of number and depth of the photometric bands. For detailed information on the depth and coverage of the optical and near-infrared data in the ECDFS, the reader is referred to Table 2 of Simpson et al. (2014).

Since radio identification has been proven to be an efficient tool to search for counterparts of bright SMGs (e.g., Ivison et al. 2002; Chapman et al. 2005; Hodge et al. 2013), we first match our SMGs to the radio source catalogs. As shown in Figure 1, 714 of 716 ALMA maps in the UDS field are covered by the available VLA observations. There are 404 radio sources (Figure 2) that fall inside the 174 diameter FWHM of the primary-beam coverage of the 714 ALMA maps. To identify probabilistic radio counterparts to the low-resolution, SCUBA-2-detected submillimeter sources, we include all 404 ≥4σ radio sources within the ALMA maps in our analyses.

Zoom In Zoom Out Reset image size

Before matching ALMA SMGs to the radio sources, we first check the cumulative number of matches to obtain an appropriate matching radius between ALMA SMGs and radio sources. A radius of 16 is chosen because the cumulative number of matches becomes flat beyond this radius. Within this matching radius, the false-match rate is ~1%. From the 695 AS2UDS SMGs, 693 are covered by the VLA radio observations. Among these, 268 ALMA SMGs match to 259 radio sources within 16 (Figure 2), with nine radio sources having two ALMA counterparts. In total, 39% (268/695) of the AS2UDS SMGs have a radio counterpart brighter than the 4σ limit of the VLA catalog.

We then assess the robustness of our ≥4σ radio catalog. As we showed above, there are 404 radio sources in the area covered by our ALMA maps. Of these, 259 radio sources are counterparts to ALMA SMGs, along with 42 radio sources that lack both K-band and ALMA counterparts (and hence may be spurious). However, using the IRAC coverage of the field, we find that 17 of the 42 have 3.5 and 4.6 μm detections, indicating that about half of these are real radio sources but lack the K and ALMA detections. This suggests that the spurious source fraction in our radio catalog is less than 25/404, or 6%. Raising the significance cut on the radio catalog to ≥5σ reduces the number of K/IRAC/ALMA blank radio sources to 10 (from 310 radio sources, or an upper limit on the spurious fraction of 3%) but also removes 40 radio counterparts to ALMA SMGs and thus reduces the completeness of our identifications. For this reason, we have chosen to retain the ≥4σ flux limit on the radio catalog.

To start with, for the SCUBA-2-detected submillimeter sources, we first consider all ≥4σ radio sources within the ALMA primary beam as potential counterparts. Then we calculate the corrected-Poissonian probability, p-value (Downes et al. 1986; Dunlop et al. 1989), for all 404 radio sources falling in our ALMA maps by using

$$\begin{eqnarray}E={P}_{{\rm{c}}} & \quad {P}^{* }\geqslant {P}_{{\rm{c}}}\\ E={P}^{* }\{1+\mathrm{ln}({P}_{{\rm{c}}}/{P}^{* })\} & \quad {P}^{* }\leqslant {P}_{{\rm{c}}},\end{eqnarray} \tag{ 1 }$$

where P_c is the critical Poisson probability level given by ${P}_{c}=\pi {r}_{{\rm{s}}}^{2}{N}_{{\rm{T}}}$, in which N_T is the surface density of the radio sources and r_s is the search radius (in this work, it is the radius of the ALMA primary beam). Then, given P* for a radio source, we can derive the probability that it is a counterpart of single-dish-detected submillimeter sources by p = {1 − exp(−E)}.

As shown in Figure 2, the fraction of counterparts of ALMA SMGs among the radio sources dramatically decreases when p > 0.065. Hence, we adopt p ≤ 0.065 as our limit for the probabilistic association of radio sources to single-dish submillimeter sources, while we consider those radio sources with p = 0.065–0.10 as "possible" identifications. Looking at all 404 radio sources falling in our ALMA maps, 41 of these have p > 0.065–0.10 and are thus only classified as "possible" counterparts (Figure 2). Of these "possible" counterparts, the vast majority (36/41) do not match to an ALMA-identified SMG. As a result, the five radio sources from these 41 that do match to ALMA SMGs within 16 are also removed by utilizing the p-value cut. We also show the spatial offset of SCUBA-2 source positions and radio sources in Figure 2. We see that those radio sources with p > 0.065 have spatial offsets larger than 55 from the nominal SCUBA-2 positions. However, if we simply adopt this smaller match radius to search for radio counterparts to SCUBA-2 sources, we will remove ~20 of the radio counterparts to actual ALMA SMGs. Therefore, in this work, we prefer to consider all radio sources within the ALMA primary beam but apply a p ≤ 0.065 cut to identify those that are likely counterparts to the SCUBA-2-detected submillimeter sources. As a result, the precision of radio identification of counterparts to single-dish-detected sources increases from 64% (259/404) to 70% (254/363) by utilizing this p-value cut. Precision is defined as the ratio between the correctly identified SMGs and the total number of predicted SMGs by radio identification/machine-learning classification.

To identify those multiwavelength properties that differentiate the SMGs from the wider field population, we define radio sources that do not match to an ALMA-detected SMG within 26 (this is conservatively chosen to be larger than our 16 matching radius) as "non-SMG" radio sources. Including the 53 radio sources within the "blank-ALMA" maps, in total, there are 137 non-SMG radio sources falling within our ALMA maps. Although, as we show later, on average, the radio sources within the "blank-ALMA" maps have faint submillimeter emission, we put them into the sample of non-SMGs for simplicity before we perform the stacking analysis. We will discuss the properties of radio sources that are counterparts of SMGs and non-SMGs in Section 4.

To develop a method to differentiate SMGs and non-SMGs using multiwavelength data, we adopt the UDS DR11 photometric-matched near-infrared/optical catalog (W. Hartley et al. 2018, in preparation) to identify counterparts and measure the near-infrared/optical colors of SMGs.

As we described above, only those sources within the overlapped region of UKIDSS and IRAC have sufficient photometric coverage and estimated photometric redshifts, as well as absolute magnitudes, which we will use in our machine-learning method. Hence, we limit our identification of counterparts to the ALMA SMGs in this region. In total, 607/716 ALMA maps fall in this region, and 583/695 ALMA SMGs are detected within these maps with K ≤ 25.9 mag.

To select a suitable matching radius between K-band galaxies and ALMA SMGs, we test radii between 05 and 10 in steps of 01 and match the K-band galaxies with the ALMA SMGs. At each step, we randomly offset the K-band galaxies in right ascension or declination by 10''–20'' to estimate the false-match fraction as a function of matching radius. At a match radius of 06, 514 K-band galaxies from the UKIDSS DR11 photometric catalog match to ALMA SMGs with a false-match fraction of ~3.5% (~18 false matches). A match radius of 05 reduces the false-match fraction to 2% (~10 false matches) but also reduces the total number of matches by 20. A larger match radius increases the matched sources, but the new matches are dominated by false matches. Therefore, we adopt a match radius of 06.

In the overlap region of UKIDSS and IRAC, there are 483 K-band galaxies that match to ALMA SMGs within our adopted 06 matching radius. We show the number and fraction of multiwavelength counterparts of ALMA-detected SMGs in Figure 3. We find that ~83% (483/583) of the ALMA SMGs have K-band counterparts, but the number of counterparts dramatically decreases at bluer wavelengths due to their dusty nature (and their likely high redshifts). For the optical and near-infrared data, we use the 3σ limits to identify the counterparts as shown in Figure 3. Because of the relatively low resolution of the IRAC data, a more conservative 5σ cut is adopted for identifying counterparts and measuring colors in these bands. Figure 3 also presents the number of SMGs that have photometric redshifts, which are estimated based on DR11 photometric catalog, and hence have absolute H-band magnitudes available to be used in the following analyses.

Zoom In Zoom Out Reset image size

The details of the identification of radio and optical/near-infrared counterparts to the ALESS SMGs in the ECDFS field are presented in Hodge et al. (2013) and Simpson et al. (2014), respectively. Out of the 96 ALMA SMGs, 45 have radio counterparts (Hodge et al. 2013). Simpson et al. (2014) measured aperture photometry in 19 wavebands for the 96 ALMA SMGs. Among these, 77 are securely detected and have sufficient photometry to derive a photometric redshift and estimate the rest-frame H-band absolute magnitudes.

For the single-dish-detected submillimeter sources, we first use the IRAC-based photometric catalog of sources in the ECDFS from Simpson et al. (2014) to match 88 LESS submillimeter sources (Weiß et al. 2009) for which there are good-quality ALMA maps from Hodge et al. (2013). We include in this the 19 submillimeter sources for which the corresponding ALMA map detected no SMGs (the "blank-ALMA" maps). In total, there are 323 IRAC-detected galaxies located within the 88 ALMA primary beams. We will use these galaxies to test our methodology in the following analysis.

Having selected five properties that are likely to have the diagnostic power to differentiate SMGs from the non-SMGs, we first use the SVM model to build a nonlinear classifier for optimally separating these two populations. This is implemented by using the algorithm coded in the scikit-learn¹⁸ Python package (Pedregosa et al. 2011). The SVC takes a labeled training set (in this case, "SMG" versus "non-SMG") and associated set of feature vectors (e.g., observable colors) and attempts to build hyperplanes that maximize the separation between the two classes in the n-dimensional (in this case, n = 5) feature space. Having established the hyperplane(s), new, unlabeled test data can be presented to the trained classifier to determine which class they belong to according to their relative position in this five-dimensional parameter space.

We note that the classification cannot be performed using the SVC if an object has a missing feature. This occurs if we have only a limit on the color of (J − K) or ([3.6]–[4.5]) due to the lack of a secure detection at the J band or 4.5 μm. Unfortunately, there are a number of possible causes for the lack of J or 4.5 μm detection, including dust reddening, geometry, star formation history, and redshift. Therefore, we prefer not to predict these missing values through the statistical imputation algorithms (e.g., Pelckmans et al. 2005). Instead of mixing the observable properties with predicted values, we test the influence of sources with missing values using a second machine-learning model, XGBoost, which has the capacity of performing classification with missing values.

We then train the SVM classifier through a training set that includes ALMA SMGs and non-SMG K-band galaxies, which have the secure measurement of five selected properties within the ~50 arcmin² area covered by our ALMA maps in the UDS. In total, 334 ALMA SMGs and 1271 non-SMGs that have secure measurements of our five selected properties are utilized to construct the training set.

We optimize the classifier parameters via k-fold cross-validation (Kohavi et al. 1995). Here we use k = 5; i.e., we randomly divide the training set into five equally sized "folds." The classifier is trained on k − 1 folds and validated on the remaining fold. We use the recovery rate (also called the true-positive rate (TPR), recall, or sensitivity in statistics), FPR (also referred to as the false-alarm rate or 1−specificity), and precision (also called positive predictive value) as the evaluation metrics to optimize the parameters of the SVM classifier. The recovery rate is the ratio between the number of correctly classified SMGs and the total number of ALMA SMGs in the data set. The FPR is the number of objects incorrectly classified as SMGs over the total number of non-SMGs in the data set. We defined the precision in Section 2.3.1 as the ratio between the number of correctly identified SMGs and the total number of predicted SMGs by the classifier. An optimized classifier will maximize the recovery rate and precision while simultaneously minimizing the FPR.

The SVM classifiers use a "kernel" to efficiently compute the dot product between two vectors in feature space (i.e., a similarity measure) and build a decision function that is analogous to defining a "decision" energy resulting from placing a kernel at the position of the observed properties of a source (Cristianini & Shawe-Taylor 2000). The fivefold cross-validation shows that the most efficient kernel function for separating SMGs from K-band-detected galaxies is the polynomial kernel, which is defined as

$$\begin{eqnarray}&&k(x,x^{\prime} )={(\gamma (x\cdot x^{\prime} )+{c}_{0})}^{d},\end{eqnarray} \tag{ 2 }$$

where x and x' represent feature vectors in the input space, (x  x') is their inner product, d denotes the degree of the polynomial kernel function, and c₀ is a constant coefficient that is an independent parameter in kernel function. The other two parameters of the SVM algorithms with a polynomial kernel are γ and C, where γ represents the adjustable kernel width parameter, which is responsible for the topology of the decision surface, and C sets the width of the margin separation of different classes of objects (e.g., Małek et al. 2013; Kurcz et al. 2016). The fivefold cross-validation shows that the default value of these parameters in the scikit-learn package, C = 1.0, γ = 1/n features (here n = 5), d = 3, and c₀ = 0.0, are optimized for performing the classification in our work by an SVM classifier with a polynomial kernel.

The feature selection module in the scikit-learn package can also select the best features for classification based on univariate statistical tests (Pedregosa et al. 2011). The univariate score is derived by U_score = −log(p), where p is the p-value of the corresponding univariate feature (Pedregosa et al. 2011). Among the five features we selected, the best one for separating SMGs from non-SMGs is (J − K) color with a U_score = 891, followed by ([3.6]–[4.5]) color with a score of 707, (K − [3.6]) with a score of 695, and the absolute H-band magnitude with U_score = 579. The photometric redshift has a relatively lower univariate score of 324; however, as we described above, including photometric redshift and absolute H-band magnitude as the input features for machine learning does not affect the completeness of our analyses but increases the recovery rate of the SVM classifier by about 6%.

The sample we used for performing the SVM machine-learning classification are K-band-detected galaxies that have secure measurements of all five selected properties. To increase the completeness, we also include objects that lack a secure detection at the J band (i.e., have a limit on their (J − K) color) or at 4.5 μm (i.e., have a limit on the ([3.6]–[4.5]) measurement). This increases the sample size of the training set from 1605 to 1832, in which 366 are ALMA SMGs and 1466 are non-SMGs. The training set we use in our analysis is given in Table 1.

As a test of the efficiency of the SVM classifier, we have also applied a second machine-learning classifier to our sample. This is XGBoost¹⁹ (Chen & Guestrin 2016), which is a scalable machine-learning system for tree boosting. In this tree ensemble model, the input features will be first divided into different "leaves." Then, the algorithm computes the optimal weight of each "leaf" and calculates the corresponding optimal value, which will be used as a quality score of a tree structure. The structure of a tree is built by a greedy algorithm that starts from a single leaf and iteratively adds branches to the tree. Instead of enumerating all possible tree structures, XGBoost first calculates a gain of a "leaf." If the gain of the corresponding leaf is smaller than the minimum loss reduction (γ), the branch will not be added to the tree. One of the key problems in tree classifiers is how to find the best split at each node (in this case, "SMG" versus "non-SMG"). XGBoost finds the best solution among all possible splits based on the aggregated statistics according to percentiles of feature distribution. For the missing value, XGBoost classifies the instance into the optimal default direction that is learned from the data. The input properties of unlabeled test data will be divided into the same leaves as the training set, and the final prediction will be calculated by summing up the score in the corresponding "leaves" of a test object (Chen & Guestrin 2016).

Similarly, we optimize the parameters of the XGBoost tree classifier via the fivefold cross-validation. Unlike the SVM implemented in the scikit-learn package, which directly predicts the class label of an object, the XGBoost classifier estimates a probability of an object being an SMG. We then also use the area under the curve (AUC) of a receiver operating characteristic (ROC) curve (Fawcett 2004), as well as the assessment metrics—recovery rate, precision, and FPR—we used before to optimize the parameters of the XGBoost classifier. The ROC curves are constructed by comparing the recovery rate against the FPR, as the probability threshold is varied. Typically, an AUC higher than 0.9 indicates an excellent classifier (e.g., Lochner et al. 2016).

For boosting trees, we find that a learning rate η = 1.0 and a maximum number of iterations num_round = 5 are enough for performing a good classification (AUC > 0.9). The other two parameters for a binary classification are the minimum loss reduction (γ), which is required to make a further partition on a leaf node of the tree and the maximum depth of a tree. The fivefold cross-validation indicates that γ = 1.0 and a maximum depth of 6 are the optimized parameters for the XGBoost classifier. An object is classified as an SMG if the probability ≥0.5.

For both machine-learning algorithms, we use a uniform weight for all objects and properties. We repeat the fivefold cross-validation 100 times, calculate the median and standard deviation of each metric, and present the values of these metrics for the optimized classifiers in Table 2.

To test the efficiency of our machine-learning method, first we carry out a "self-test," i.e., using all K-band galaxies within the ALMA primary beams to build a test set. The K-band galaxies in the 108 "blank-ALMA" maps are also included in the test sample, since it is not possible to know a priori which submillimeter sources will have "blank-ALMA" maps (i.e., contain no SMGs above a 4.3σ significance cut) when we identify counterparts for single-dish-detected submillimeter sources. In total, 2033 K-band galaxies lie within the ALMA primary beams and have secure measurements of five selected properties; 363 of these are in "blank-ALMA" maps. We then utilize the training set and SVM model to identify the likely SMGs in this test sample and compare this to the actual catalog of ALMA-detected SMGs in these maps.

We present the results of the "self-test" in Figure 5. The SVC classifies 378 counterparts as "SMGs" from the 2033 K-band-detected galaxies within the ALMA primary beams, somewhat more than the 334 ALMA-detected SMGs in these fields. For the 334 ALMA-detected SMGs with all five features, 252/334 (75%) are recovered by the SVC model. The precision of this machine-learning method is therefore 67% (252/378). We note that this is a lower limit on the precision for the machine learning, since we consider all K-band galaxies in the "blank-ALMA" maps as non-SMGs. However, our stacking of far-infrared observations shows that there are faint SMGs present in the "blank-ALMA" maps, and some of machine-learning method classified "SMGs" in the "blank-ALMA" maps will be true counterparts of SMGs that are marginally too faint to be detected by ALMA (as we show later). The results of the fivefold cross-validation shown in Table 2 indicate that the precision would increase to 82% if we had excluded the "blank-ALMA" maps from the analysis.

Zoom In Zoom Out Reset image size

As shown in Figure 5, those galaxies that are classified as "SMGs" by the SVM classifier but not detected by ALMA at >4.3σ (typically S_870μm ≥ 0.9 mJy) have very similar properties to the ALMA-detected SMGs; i.e., they are red in the near-infrared, at high redshift, and bright in the rest-frame H band. We will discuss the properties and the results of stacking the ALMA maps at the position of these galaxies in Section 4. We also note that the SMG counterparts that are not recovered by the machine-learning code tend to be those at lower redshifts, which are faint in the rest-frame H band or blue in their near-infrared colors.

We also highlight in Figure 5 the K-band galaxies that have radio counterparts with p-values p ≤ 0.065. As we described in Section 2.3.1, we use the p-statistic to identify radio counterparts for single-dish-detected submillimeter sources. For the 2033 K-band galaxies in the UDS test sample, 235 also have >4σ radio detections with p ≤ 0.065. Among these, 167/235 (71%) are matched to ALMA-detected SMGs within 16. Therefore, half of the 334 ALMA SMGs are recovered by radio identification alone. Combining the machine-learning classification with the radio identification, 285/334 (85%) of the ALMA SMGs are recovered with a precision >62%. This proves that our combined radio and machine-learning method can efficiently recover SMGs from the general population of K-band-selected galaxies.

To increase the completeness of the self-test sample, we also include the K-band-detected galaxies that lack a secure detection at the J band or 4.5 μm and adopt the XGBoost machine-learning module to perform the classification. The sample size is increased to 2305, with 366 being ALMA-detected SMGs. The XGBoost model identifies 409 "SMGs" from this enlarged test sample. For the ALMA SMGs, 270/366 (74%) are recovered with a precision of >66%. Combining with the radio identification, 310/366 (85%) of ALMA SMGs have been recovered with a precision of >62%.

We note that the performances of the two machine-learning modules are very similar according to the fivefold cross-validation and this self-test (Table 2). To keep the consistency with Figure 5, we show the analyses of the SVM classification in the following figures and use machine learning to refer to the SVM method, unless we explicitly state that we are using XGBoost.

We expect the "self-test" will provide an overly optimistic indication of the success rate of our method, as it uses the same sample for both the training and testing. For that reason, we also undertake a number of independent tests, which use distinct samples for the training and testing.

First, we divide the AS2UDS sample into independent halves to test our method, which we will term a "half–half" test. We randomly assign the K-band galaxies in half the ALMA maps to the training set and use the galaxies within the other half of the ALMA maps as the test sample. We then utilize this training set and our combined radio and SVC machine-learning method to classify the likely counterparts of SMGs in the independent test sample. We repeat this "half–half" test 100 times to estimate the scatter in the recovery rate and precision. The median recovery rate of the combined radio and machine-learning method of these tests is 86% ± 3% with a median precision of 61% ± 3% (Figure 6). This "half–half" test confirms the success of our method when used to identify the counterparts of SMGs using a training set with similar photometric coverage and depth. This success rate is therefore expected to be representative of what will be achieved when we apply our method to identify the SMG counterparts in the S2COSMOS survey (J. M. Simpson et al. 2018, in preparation).

Zoom In Zoom Out Reset image size

The next independent test we perform is to apply the trained SVM classifier to the independent sample of ALMA-identified SMGs in the ECDFS field from the ALESS survey (Hodge et al. 2013; Simpson et al. 2014). As we described in Section 2.3.3, there are 323 IRAC-selected galaxies located within the 88 ALMA primary beams. Of these galaxies, 232/323 have secure measurements of the five selected properties that are used in our SVM classification. Among these, 47/232 sources match to ALESS MAIN sample SMGs within 15.

We show the results from applying the SVM classifier trained on the AS2UDS sample to the identification of the SMGs in ALESS in Figure 6. The recovery rate of the machine learning is 62% with a precision >71%. As we have included the 19 "blank-ALMA" maps in our test sample (which includes some galaxies classified as non-SMG but that are actually just below our submillimeter flux limit), we believe this precision is a lower limit. We also match these 232 IRAC-based galaxies with the radio catalog from Miller et al. (2008). Again, the radio identification can recover half of the ALMA SMGs with a precision of 75%. Hence, combining the radio identification and machine-learning classification, we recover 72% of the ALMA SMGs with a lower limit on the precision of 65%.

We note that the recovery rate for the ALESS sample is lower than that achieved in either the "self-test" or "half–half" test on the AS2UDS sample. To understand the cause of this, we also carry out a "self-test" on the ALESS sample (i.e., we use the ALESS SMGs and non-SMGs as both the training and analysis samples) and find that the recovery rate of classification increases from 72% to 79%, while the precision increases to 69%. The recovery rate of ALESS SMGs is still lower than that of the "self-test" on the AS2UDS SMGs.

It may be that the lower success rate for the ALESS sample is simply due to small-number statistics: the number of ALMA maps in ALESS is only 12% (88/716) of that in AS2UDS. We can test this using AS2UDS by selecting test samples of galaxies from 88 randomly selected ALMA maps from the AS2UDS survey and determining the variation in the recovery rate and precision between these test samples. We call this the "12% test," and we repeat this test 100 times to obtain the scatter. The median recovery rate of our combined radio and machine-learning method for a sample of galaxies in 88 ALMA maps is 86% ± 5%, with a lower limit on the precision of 63% ± 7% (Figure 6). When we compare the results with the "half–half" test, we find that the smaller sample size causes larger uncertainties in the machine-learning classification. We also note that four of these 100 tests have a recovery rate as low as that of the ALESS SMGs but with a range of precisions. Therefore, it appears that a recovery rate and precision as low as those seen for the ALESS test sample are possible, simply due to the small sample size. However, we note that there are also potential astrophysical reasons for the different success rates. In particular, the ALESS SMGs are typically fainter at 870 μm, with a median flux density of S_870μm = 2.2 mJy, compared to the AS2UDS SMGs, S_870μm = 3.8 mJy. As shown in Figure 7, the recovery rate of the combined radio and machine-learning method is higher for brighter SMGs. And we note that the beam of LABOCA/APEX, which is the basis of ALESS, is larger than that of SCUBA-2/JCMT used for AS2UDS, which will reduce the precision of identification of counterparts to single-dish-detected submillimeter sources.

Zoom In Zoom Out Reset image size

Thus, we argue that the decrease in the recovery rate when using the AS2UDS training set applied to the ALESS sample is probably partly caused by the relative faintness of the SMGs and larger beams of the single-dish survey in the ECDFS field. The difference between a K-selected training set in UDS and the IRAC-selected test sample in ECDFS may also affect the results of our method.

We also undertook these same tests using the XGBoost machine-learning classifier. The two machine-learning modules have a very similar performance on the "half–half" and "12%" tests, while the XGBoost classifier gives a marginally higher recovery rate (64%) with a relatively lower precision (65%) for the ALESS sample.