SPECTROSCOPY OF LUMINOUS z > 7 GALAXY CANDIDATES AND SOURCES OF CONTAMINATION IN z > 7 GALAXY SEARCHES^*

The z > 7 universe holds the keys to many open questions about galaxy formation and evolution. Cold dark matter (CDM) models predict that galaxies assemble through merging of dark matter halos (Springel et al. 2005; Cole et al. 2000; Kauffmann et al. 1993), and the distribution of initial halo masses is strongly constrained by the cosmic microwave background (CMB) measurements (Bennett et al. 2003). However, populating the dark matter halos with galaxies is done in a semi-analytic manner with many free parameters that are tuned to observations. At z ≃ 7 the universe was less than a billion years old, leaving little time for galaxies to evolve, so the luminosities and masses of galaxies at z > 7 are a direct probe of the assumptions underlying galaxy formation scenarios in structure formation models.

Advances in red-sensitive charge-coupled devices (CCDs) have allowed studies of the universe out to z ∼ 7 (e.g., Ouchi et al. 2009; Ota et al. 2008; Iye et al. 2008); however, accessing higher redshifts is difficult due to the lack of wide-field near-infrared cameras and high sky backgrounds beyond 0.8 μm in the case of ground-based telescopes. Ultra-deep studies from space with Near Infrared Camera and Multi-Object Spectrometer (NICMOS) and Advanced Camera for Surveys (ACS) on the Hubble Space Telescope (HST) have constrained the faint end of the luminosity function (Bouwens et al. 2008; Oesch et al. 2010) with deep ground-based studies and NICMOS parallel surveys placing useful limits on its bright end (Stanway et al. 2008b; Mannucci et al. 2007; Henry et al. 2009), and lensing surveys also providing useful constraints on fainter galaxies (Kneib et al. 2004; Richard et al. 2008; Bradley et al. 2008; Bouwens et al. 2009) albeit with significant uncertainty due to the small areas covered. Unfortunately, the selection function is complicated due to contamination from faint galaxies at 1.4 < z < 2.5 where the galaxy population is also poorly understood and the fact that no objects have been spectroscopically confirmed (Stanway et al. 2008b).

In this paper, we use deep NIR imaging of the 2 deg² Cosmic Evolution Survey or COSMOS field with the Wide-Field Infrared Camera (WFCAM) instrument on the United Kingdom Infrared Telescope (UKIRT) and the Wide-Field Infrared Camera (WIRCAM) instrument on the Canada–France–Hawaii Telescope (CFHT) along with deep Spitzer Infrared Array Camera (IRAC) and Multi-Band Imaging Photometer (MIPS) imaging to color select candidate z-dropout galaxies. We find three candidates brighter than J_AB < 23.5, which have 0.9–8 μm spectral energy distributions (SEDs) consistent with fainter z > 7 candidates discovered in the GOODS and UDF fields (Bouwens et al. 2008; Stanway et al. 2008b; Oesch et al. 2010) and have mid-infrared (MIR) colors consistent with the selection proposed in Stanway et al. (2008b). We then present deep spectra of these objects obtained with Keck-DEIMOS in the 0.64–1.02 μm range and Keck-NIRSPEC in the 0.94–1.10 μm and 1.52–1.80 μm ranges. In addition to these three candidates, we find over 3500 objects which are at z ∼ 1.8 and would be selected as z > 7 objects, were it not for the large difference in depth between the near-infrared and optical bands. Using this sample of interlopers, we discuss the sources of contamination for z > 7 studies and strategies for removing them.

Throughout this paper, we use AB magnitudes or fluxes in units of Jansky and a Ω_m = 0.3, Ω_v = 0.7, H₀ = 70 km s⁻¹ Mpc⁻¹ cosmology.

The 2 deg² COSMOS field is covered by a combination of 30 broad, intermediate, and narrow ultraviolet (UV), optical, and NIR filters including data from GALEX, CFHT, Subaru, HST, and Spitzer. These observations are described in detail in Zamojski et al. (2007, GALEX UV), Capak et al. (2007, 2008, Subaru, CFHT, and UKIRT optical and NIR), Koekemoer et al. (2007) and Scoville et al. (2007, HST optical and NIR), and Sanders et al. (2007, Spitzer near- and mid-infrared). In addition, there are extensive, X-ray data obtained with the Chandra X-ray Observatory (Elvis et al. 2009) and XMM-Newton (Hasinger et al. 2007), submillimeter data from the IRAM 30 m (Bertoldi et al. 2007), along with 20 cm (Schinnerer et al. 2007) and 90 cm (V. Smolcic et al. 2009, private communication) radio images from the Very Large Array (VLA).

Throughout this paper, we follow the filter notation used in Capak et al. (2007). In particular Subaru–Sloan filters are noted with the superscript ⁺ (e.g., z⁺) and CFHT Megaprime filters are denoted with the superscript * (e.g., u*) to differentiate them from each other and the true Sloan Digital Sky Survey (SDSS) filters which have somewhat different bandpasses.

This paper also makes use of new deep J-band data obtained with the WFCAM instrument on UKIRT (Hirst et al. 2006), along with H and K_s data obtained with WIRCAM on CFHT (Puget et al. 2004) telescopes as part of the COSMOS survey (McCracken et al. 2010). The J, H, and K_s data were taken over the course of three years with over 500 exposures at each position. This is more than sufficient to remove cosmic rays, moving objects, satellites, and other transient objects that may be mistaken for high-z objects. Electronic cross-talk artifacts near bright objects are a significant problem in all three data sets. The nature of the cross talk in the H and K_s data allow the artifacts to be modeled and removed in all but the most severe cases (McCracken et al. 2010). Any remaining cross talk is easily identified because it consists of 64 equally bright/dark images at 64 detector pixel intervals in Declination. Cross-talk artifacts are still present in the UKIRT WFCAM data. In WFCAM data, the cross-talk artifacts occur at 128 detector pixel intervals in either right ascension or declination from bright objects depending on position within the array and can be identified by their distinct positive–negative signature.

A K_s-band selected object catalog was generated with SExtractor (Bertin & Arnouts 1996) using the CFHT–WIRCAM K_s image for detection and point-spread-function-matched images for photometry as described in Capak et al. (2007). The SExtractor detection settings are those used in McCracken et al. (2010). The K_s-band image was chosen over J band for detection because while the image depths are similar, the image quality is significantly better in the K_s-band images. Moreover, the expected object colors mean they should be equally bright or brighter in K_s than J. In addition to a detection in K_s band, we also require a 5σ measurement in a 3'' aperture for both the J and K_s bands for an object to enter our analysis. This selection improves the robustness of our object identification, removing a significant number of artifacts which are only present in either the J- or K_s-band images. The resulting catalog has a 5σ limiting magnitude in a 3'' aperture of 23.7 on the AB system and is 80% complete at this flux level for point sources (McCracken et al. 2010).

The H-band data were not available when the objects were selected, but is used in later analysis. The photometry was measured in the same way as the for the J and K_s bands and has a 5σ limiting magnitude in a 3'' aperture of 24.0 on the AB system.

As our primary selection criteria we require: J < 23.7, z⁺ − J ⩾ 1.5, J − K_s > 0, K_s − 4.5 μm >0, no bad pixels or cross talk in the z⁺ or J bands, and less than a 2σ detection in the B_J, g⁺, and V_J bands. This selection removes many galactic stars which have blue J − K_s and K_s − 4.5 μm >0 colors and most low-redshift galaxies which are detected in the optical bands. However, our sample still contained over 3500 objects, many of which appear to be dusty or passive z ∼ 2 galaxies or galactic L and T dwarf starts based on their SEDs (see Section 5). Postage stamps in all 22 bands and a weighted average combination of the 0.3–0.8 μm bands were visually inspected to search for low level optical flux, resulting in a sample of 22 objects with no clear optical detections. Of these 22 objects, 3 had flat SEDs from 1.2–4.5 μm, consistent with z > 7 objects. The remaining 19 candidates have very red SEDs implying they could be dusty z ∼ 2 galaxies. The positions of the objects are given in Table 1 and the F814W, and z⁺ limits along with measured fluxes for the 3 objects selected for follow-up are tabulated in Table 2. The 2σ limiting fluxes in a 3'' aperture for the other broadband images are u*>27.3, B_J > 27.9, g⁺ > 27.2, V_J > 27.1, r⁺ > 27.3, and i⁺ > 26.9. The intermediate and narrow bands have limiting magnitudes ∼1–2 mag brighter than the broad bands (Ilbert et al. 2009).

Table 1. Positions

ID	R.A. (J2000)	Decl. (J2000)	FWHM ('')	Stellarity
1	10h01m36317	+1:37:00.42	2.5	0.98
2	9h59m25507	+2:21:27.79	1.5	0.96
3	9h59m14609	+2:27:36.57	2.0	0.53

Download table as:  ASCIITypeset image

The color–color plot for the selected objects are compared with those from the literatures in Figure 1. Postage stamps of the ground-based 0.3–2.2 μm images along with Spitzer IRAC Ch1 (3.6 μm) and Ch2 (4.5 μm) are shown in Figure 2. HST-ACS stamps are shown in Figure 3. All three objects have full width at half-maximum (FWHM) measurements in the K_s band which are larger than that measured for nearby stars. However, only object 3 is inconsistent with a star based on the SExtractor stellarity index which uses a neural network to consider the effects of signal-to-noise ratio (S/N) and confusion on the size measurements. Object 1 is near several foreground, optically detected sources which partially confuse the IRAC photometry for this source. In the 3.6 and 4.5 μm bands fluxes were recovered using 14 diameter apertures centered on the K_s-band position and subsequently correcting for the missing flux. However, the fluxes at 5.8 and 8.0 μm could not be reliably measured without significant de-convolution. Objects 2 and 3 are isolated. Although object 2 is technically detected in the ground-based i⁺ image we included it in our sample because the detection is marginal and it is not detected at >2σ in the ACS F814W data or the z⁺-band data.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

None of the sources is detected in the XMM-Newton 0.5– 10 keV band X-ray data at a limit of 7.26 × 10⁻¹⁶ erg s⁻¹ cm⁻². Sources 2 and 3 fall into a region with available Chandra 0.5–10 keV band X-ray imaging and are not detected at a 3σ upper limit of 7.4 × 10⁻¹⁶ and 1.0 × 10⁻¹⁵ erg s⁻¹ cm⁻², respectively.

Source 2 is detected in the deep MIPS GO-3 imaging of the cosmos field with a flux of 66 ± 12 μJy at 24 μm (LeFloc'h et al. 2009; Sanders et al. 2007). Sources 1 and 3 are undetected in the GO-3 MIPS 24 μm images. Postage stamps of the Spitzer MIPS 24 μm imaging are shown in Figure 4.

Zoom In Zoom Out Reset image size

Sources 2 and 3 fall into a region with deep MAMBO (Bertoldi et al. 2007) imaging at 1.2 mm, and neither is detected with fluxes >2 mJy. Finally, none of the sources are detected in the VLA 20 cm data with an rms of 16 μJy for object 1 and 10 μJy for objects 2 and 3 (Schinnerer et al. 2007) or at 90 cm with an rms of 0.53, 0.42, and 0.47 mJy for objects 1, 2, and 3, respectively (V. Smolcic et al. 2009, private communication).

Near-IR spectra were obtained on 2008 February 22 and 23 under photometric conditions with 05–07 seeing using NIRSPEC on the Keck-II telescope (McLean et al. 1998). The data were collected in low-resolution long-slit mode using the 076 slit with the N1 and N5 order blocking filters. The resulting spectral resolution was R ∼ 1500 with a pixel scale ∼1.6 Å and ∼2.7 Å in the N1 and N5 setting, respectively, and a spatial pixel scale of 018. The grating was tilted to cover 0.94–1.10 μm and 1.52–1.80 μm in N1 and N5, respectively. These ranges were chosen to cover Lyα (1216 Å) at 6.7 < z < 8, [O ii] (3727 Å) at 1.5 < z < 2, and Hα (6563 Å) at 1.3 < z < 1.8. A nearby bright star was used to guide the acquisition. For objects 2 and 3 the slit position angle was set, so we could observe the alignment star and the object simultaneously in at least one of the dither steps. The data were taken in an "ABBA" dither pattern with a 15'' throw along the slit. The integration times for objects 1, 2, and 3, respectively, are 2 hr, 3 hr, and 3.75 hr in N1 and 1 hr for all objects in N5. The data were reduced using the NIRSPEC reduction package described in Becker et al. (2006). Once reduced, the slits were rectified and co-added using a bright object positioned on the slit to register the images.

Latent images from prior observations can be a problem in NIRSPEC data. To ensure these were not affecting the final results we inspected each frame for latent images. We also split our data by night, dither pattern, and dither position, inspecting each of these stacks for latent images which appeared in one frame but not others. Latent images from a bright standard used to determine the slit function were only visible in the first dither N1 dither pattern of the spectra taken on object 3 on February 22 and the first N1 exposure taken on object 3 on February 23. The latents were not at the expected object position.

Optical spectra were obtained for objects 2 and 3 with DEIMOS on the Keck-II telescope (Faber et al. 2003) on 2008 November 23 under partially cloudy conditions and 2008 November 25 under clear conditions with ∼1'' seeing. The data were collected with 10 wide slits, the 830 l mm⁻¹ grating tilted to 7900 Å and the OG550 blocker. The resulting spectral resolution was R ∼ 2370 with a pixel scale of ∼0.47 Å and a spatial pixel scale of 01185. The data were reduced with a modified version of the DEEP2 DEIMOS pipeline (Marinoni et al. 2001). In addition to the standard processing, this modified pipeline constructs and subtracts a median background and accounts for dithering, positional shifts, and photometric scaling when combining the spectra. The objects were dithered ±3'' along the slit to improve the sky subtraction and mitigate ghosting (internal reflections) from the 830 l mm⁻¹ grating. Bright objects in the mask were used to determine the amount of atmospheric extinction on November 23, and integrations with less than 90% throughput were removed. Each exposure was 30 minutes for a total of 4.5 hr of unextincted integration time.

The positions of the NIRSPEC and DEIMOS slits are shown in Figure 5 and the two-dimensional spectra are shown in Figures 6, 7, and 8. We tabulate the range in line sensitivity assuming a 10 Å extraction box in the spectral domain, comparable to the expected observed frame line width, and the limits this places on the amount of star formation in Table 3.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We conducted a close inspection of the spectra, including smoothing and binning by up to 32 pixels (∼115 Å) in the dispersion direction and 10 pixels (1'') in the slit direction. Since the visible spectral features are weak, we used two methods to estimate their significance. First, we created a variance map including read noise, Poisson noise, estimated errors in background subtraction, variations in depth from bad pixels and dithering, and noise correlation due to re-sampling. This map was used to estimate the significance of any feature. As a second estimate of significance we used SExtractor to search for low-S/N peaks in the two-dimensional spectra with the object and alignment star positions masked. Using this catalog of noise peaks, we then calculate the probability that a peak of a given flux would fall within ±1'' of the object position.

At the expected position for object 1, a weak line is visible at 1.0563 μm in the N1 spectra. The significance of this feature is 2.4σ based on the noise map and 2.6σ based on the noise peak method. If this line is Lyα it places the object at z = 7.69; however, the line could also be [O ii] at z = 1.83. Nothing is seen in the N5 spectra of object 1 at the expected position. However, a line is seen at 1.560 μm at the position of a nearby optically detected foreground object seen in Figure 2 and noted in Section 2. The noise map yields a significance of 3.8σ for this feature and no other peaks at this flux level are found in the spectra. This would place the neighboring object at z = 1.37 if the line is Hα, which is the solution favored by the photometric redshift from Ilbert et al. (2009), but [O ii] at z = 3.19 cannot be ruled out. [O iii] (5007 Å) or Hβ (4861 Å) lines at z ∼ 2.2 are unlikely for both objects because we would expect to see both lines in the spectra. However, we note there is significant scatter in the observed [O iii] and Hβ line ratios so this redshift cannot be entirely excluded at our S/N.

Nothing is detectable in the DEIMOS, N1 or N5 spectra of object 2.

The DEIMOS spectra of object 3 shows a feature at 0.9672 μm with a 3.4σ significance based on the noise map and no other features with equal significance found in the spectra. This suggests a redshift of 6.95 if the line is Lyα, assuming the blue edge is at the line center. However, this line could also be [O ii] at z = 1.60 or less likely Lyβ at z = 8.43. Nothing is detected in either the N1 or N5 spectra for object 3. The sensitivity of the N1 spectra is ∼6× lower than the DEIMOS spectra at ∼0.9672 μm so the non-detection does not rule out the line seen in the DEIMOS spectra. The sensitivity in the N5 spectra is also insufficient to detect Hα at z = 1.60 for the typical Hα to [O ii] star-forming line ratio of 1.77 (Kennicutt 1998) unless the source were obscured by A_v > 7, which is significantly larger than the value implied by the broadband colors.

To quantify the probability these sources are at z > 7 and constrain their physical properties, an SED fitting analysis was conducted using the Le Phare code.¹⁹ Both empirical (Polletta et al. 2007; Budavári et al. 2001) and theoretical (Bruzual & Charlot 2003) templates were fit with up to A_V = 3 mag of obscuration using a Calzetti et al. (1996) or Prevot et al. (1984) extinction law and an additional 2175 Å bump added as a free parameter to the extinction law (Ilbert et al. 2009). Stars from the Bixler et al. (1991) and Chabrier et al. (2000) libraries covering O to T stars were also fit to the photometry. The optical detection limits were conservatively set at 3σ rather than 1σ values since our knowledge of z > 1 SEDs are limited. The results of the SED fitting analysis are shown in Table 4 and Figure 9.

Zoom In Zoom Out Reset image size

The J-band flux is critical in the SED fitting analysis because it constrains the break strength between the z⁺ and J bands, which in turn constrains the redshift (Stanway et al. 2008b; Bouwens et al. 2008). However, this is mitigated somewhat by the high-quality Spitzer IRAC and CFHT H-, K_s-band data which also place strong constraints on the SED shape. To test for the effects of photometric offsets we introduced a shift of ±0.5 mag in J band and removed it and the IRAC 5.8 and 8.0 μm bands from the fits.

The best fit for object 1 is an obscured (A_v = 1.2) star-forming galaxy or unobscured active galactic nucleus (AGN) at 7.1 < z < 8.9, in good agreement with the potential line at z = 7.7. The star formation rate based on the SED fit would between 190–3000 M_☉ yr^-1, depending on the obscuration correction, so this source may be detectable in the millimeter if it is a star-forming galaxy. If instead the source is a Type 1 AGN, the X-ray flux limits the black hole mass to < 6 × 10⁷ M_☉ at z ∼ 7 based on an analysis of Type 1 quasars in the COSMOS field (Lusso et al. 2010).

Removing or decrementing the J-band magnitude by 0.5 mag for object 1 produces photo-z solutions between 0.6 < z < 1.2 with an extinguished star-forming galaxies. However, all of these fits imply >3σ detections in the i⁺ and F814W bands and >2σ detections in the z⁺ band. Furthermore, they suggest star formation rates of 30–1000 M_☉ yr^-1, which should lead to an MIPS 24 μm detection (Rieke et al. 2009). As a result these low-z solutions are strongly disfavored by the data and it is likely that this object is at z > 7.

For object 2 the low-S/N i⁺- and z⁺-band detections and 24 μm detection firmly place it at low redshift. The best-fit SED is a very young, obscured (A_V = 1.55) source at z = 1.59 with 15 M_☉ yr^-1 of star formation, in broad agreement with the 24 μm flux.

The best fits for object 3 place it at 6.7 < z < 7.3 with a 200 Myr old relatively unobscured (A_v = 0.3) stellar population or an AGN-type spectra. The star formation rate implied by the SED fit is ∼200 M_☉ yr^-1, comfortably within the mm detection limits. However, the weak H-band flux relative to the J and K is a quandary. This could be explained by an unusually strong 2175 Å dust absorption feature, but, evidence suggests that dust at these high redshifts does not have a strong 2175 Å feature (Stratta et al. 2007; Maiolino et al. 2004). Alternatively, the source could be an AGN at z > 8 with Lyα and C iv falling in the J band and H band covering a region of relatively line free continuum. Following this interpretation the line seen in the DEIMOS spectra would be Lyβ at z = 8.43, which is also seen in deep spectra of the z > 6 SDSS quasars (Fan et al. 2006), but would imply a very strong Lyα line flux to overcome the strong absorption from the intergalactic medium at these redshifts. If this is an AGN, the Chandra X-ray flux limits the black hole mass to < 6 × 10⁷ M_☉ at z ∼ 8.4 (Lusso et al. 2010).

Removing the J-band data point resulted in a fit at 2 < z < 4 by an obscured (A_v = 0.3) ∼200 Myr old stellar population with ∼200 M_☉ yr^-1 of star formation. However, the fits imply >3σ detections in the i⁺ and F814W bands and a >5σ detection in the H band which is not seen.

The reliability of photometric redshifts is tightly linked to how well the input SEDs represent the galaxy population. Photometric redshifts and photometric selections have thus occasionally erroneously identified exotic low-redshift objects as high-redshift systems (Henry et al. 2008; Mobasher et al. 2005; Pelló et al. 2004; Dickinson et al. 2000). For z > 7 galaxy searches this is especially worrisome in view of the large numbers of optically faint galaxies in the "redshift desert" (1 < z < 2.5; Steidel et al. 2004; Daddi et al. 2004) and late-type stars which have similar colors and poorly constrained SEDs. These contaminating objects are difficult to study for the same reasons as z > 7 objects; their strong spectral features are shifted to NIR wavelengths where the night sky is brighter, the density of night sky lines higher, and instruments have a smaller fields of view and lower sensitivity.

To understand the limitations of our selection and the SED models we selected a sample of optically bright (i⁺ < 26.5) objects from the COSMOS i-band catalog (Capak et al. 2007) which meet our z⁺-band dropout selection criteria (z⁺ − J > 1.5, J − K > 0, K_s − 4.5 μm >0). The intermediate-band data provide a low-resolution spectra (R ∼ 20) for these bright objects, allowing us to classify them and obtain reasonable photometric redshifts as shown in Figure 10. The majority of these objects have photometric redshifts in the 1.4 < z < 2.5 range (Figure 10), so a BzK diagram (Figure 11) provides information on the properties of these galaxies (Daddi et al. 2004). The BzK diagram indicates a mix of obscured star-forming and passive galaxies at z ∼ 2.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Given this contamination we investigated the J, K_s and IRAC color cuts proposed by Stanway et al. (2008b; Figure 12) which were formulated to remove passive objects at z < 2.5 and late-type stars. These cuts succeed in removing a large number of objects, but fail to remove a significant fraction of the contaminating population with the steepest spectral slopes. These objects are likely to have a steeper dust extinction curve than the locally observed value (Siana et al. 2008) or be high-redshift analogs of the dusty red galaxies seen in the A901/2 super cluster (Wolf et al. 2009).

Zoom In Zoom Out Reset image size

Another possible explanation are objects with extreme line equivalent widths such as those presented in Kakazu et al. (2007) and Hu et al. (2009). However, at z ∼ 1.8 these objects would mimic the blue J − H colors thought to be unique to young stellar populations at z > 7 (Oesch et al. 2010; Bunker et al. 2010). Furthermore, unlike other sources of contamination, these objects are likely to be more common at fainter magnitudes because they are thought to originate in low-mass, low-metallicity galaxies. However, the same strong emission lines would also mean these sources could be easily identified with deep spectroscopy, if the sources are bright enough. Based on this analysis a range of NIR and IRAC color cuts can reduce the number of contaminating objects, but not completely remove them. Furthermore, many of the z ≪ 7 objects exhibit blue the J − H colors thought to be indicative of young stellar populations at high redshift (Oesch et al. 2010; Bouwens et al. 2010; Bunker et al. 2010). Therefore, blue NIR colors and a strong break are insufficient to confirm sources. We conclude the strongest selection criteria presented here, and in the literature, are the non-detections in the i⁺ and z⁺ bands which rule out low-z objects based upon the strength and steepness of the detected break. To explore the necessary depth in i and z bands required to rule out low-redshift objects, in Figure 13 we take the photometry for sources meeting our z-dropout criteria and scale them to a J-band magnitude of zero. This figure shows a significant number of low-z objects have z⁺ − J > 2 and i⁺ − J > 3.5 colors. So, a survey must be at least 3 mag deeper in z⁺ and 4 mag deeper in i than J to adequately select z > 7 galaxies.

Zoom In Zoom Out Reset image size

Based on the limits required in i and z bands to rule out low-redshift objects, most surveys have reached their maximum depth for z > 7 searches unless deeper optical data are obtained. Assuming 2σ limits are sufficient to rule out low-z objects, the COSMOS i-band data are sufficiently deep to probe to J = 23.7, the depth of our present survey. The Hubble Ultra Deep Field (UDF; Beckwith et al. 2006) obtains 2σ depths of F775W = 31.1, and F850LP = 30.3, which is sufficient to remove contaminating objects brighter than F160W of 27.1, well matched to the present NICMOS data. Finally, the Great Observatories Origins Deep Survey (GOODS; Giavalisco et al. 2004) F775W and F850LP data reach 2σ depths of 28.9 and 28.6, respectively, sufficient for the depth of the present ground-based NIR imaging with a limit of J = 24.9 (Bouwens et al. 2007).

Relying on deep i- and z-band imaging is problematic as we probe deeper due to foreground objects which will occupy an increasing fraction of the sky. This effectively reduces the sample completeness by removing sources that are superimposed on faint foreground sources by chance. In our ground-based COSMOS images we lose >38% of the unmasked sky area to objects detected in the ground-based optical bands. The situation improves in space due to increased resolution and decreasing size of faint galaxies. Assuming the size distribution as a function of magnitude given in Jouvel et al. (2009), using the faintest magnitude bin to extrapolate to F775W = 31.1 we find that >10% of the UDF area will be lost to objects brighter than F775W = 31.1.

Therefore, the z − J, J − K, and F850LP − F110W, F110W − F160W based z > 7 search techniques used from the ground and space may have reached there practical limits. Deeper z − J color searches with Ultra-Vista, the United Kingdom Infrared Deep Sky Survey (UKIDDS) or Wide Field Camera 3 (WFC3) will likely be dominated by contamination unless significantly deeper i- and z-band data are obtained, and even then they are likely to be incomplete due to confusion with foreground galaxies. However, narrower adjacent filter sets such as that used in the WFC3-UDF (Oesch et al. 2010; Bunker et al. 2010) should decrease the amount of contamination by better differentiating the smooth roll-over in the SED of obscured and old stellar populations at z ∼ 1.8 and the sharp breaks caused by the Lyα forest at z > 7. However, objects with strong line emission will still be problematic.

6.1. Are These Objects at z > 7?

Based on the present photometric and spectroscopic data, it is unclear whether or not the objects presented are genuinely at z > 7. All three objects meet the color selections suggested by previous investigations (Stanway et al. 2008b; Oesch et al. 2009; Labbé et al. 2006; Bouwens et al. 2008). However, the 24 μm and marginal i⁺ detection of object 2 strongly suggests it is an obscured source at z ∼ 1.6. The SEDs of objects 1 and 3 are inconsistent with known z < 7 galaxies, AGN or stars in our galaxy, and the features seen in the NIRSPEC and DEIMOS spectra strongly suggest objects 1 and 3 are at z = 7.7 and z = 6.95, respectively.

The broadband colors and SEDs of these objects are indistinguishable from z > 7 candidates found in the UDF, GOODS, and other deep surveys (Oesch et al. 2009; Bouwens et al. 2008). With a selection function similar to those of Oesch et al. (2009) and Bouwens et al. (2008), we estimate a space density of 2.51 ± 1.78 × 10⁻⁷ mag ⁻¹ Mpc⁻³ for objects with M_UV = −23.85 (see Figure 14). This space density is consistent with the faint end counts of z ∼ 7 galaxies from the WFC3-UDF data (Oesch et al. 2010; Bunker et al. 2010) if the galaxy luminosity function follows the shape of the halo mass function shown as a black line in Figure 14. This is significantly higher than the space density implied by fits to smaller-area data, but not above the theoretical limit for the density of these sources.

Zoom In Zoom Out Reset image size

If these objects are at z ≪ 7, they point to a population of stars and/or 1.4 < z < 2 galaxies with unexpected colors that are contaminating our sample and the objects discovered in fainter samples may be contaminants. This population of unusual objects must be understood before a reliable z > 7 selection can be made.

We also explored the possibility that object 1 is strongly lensed rather than intrinsically bright. To do this we constructed a three-dimensional lens model with "Lens Tool" (Jullo et al. 2007; Kneib et al. 1993) using the stellar mass and photometric redshfits of the three nearby foreground objects. We assumed a dark-matter to stellar-matter ratio of 20:1 and a Navarro–Frenk–White (NFW; Navarro et al. 1997) profile scaled to the observed light profiles of the galaxies. This produced a magnification of ∼1.2, and up to ∼2 if the galaxies were several times more massive than estimated. As a result, we believe lensing magnification is an insignificant contribution.

The lack of conclusive spectra is troubling because these objects are at the practical spectroscopic limit for 8–10 m ground-based telescopes. Integration times of ∼25 hr (∼3 nights) per object would be required to reach an S/N of 5 for the features seen in these spectra. Multi-band NIR ground-based spectrographs will allow better probes by simultaneously recording spectra from a broad wavelength range, however gaps will sill remain in the spectral coverage due to atmospheric absorption. Multi-object spectrographs will not provide a significant advance because the sky surface density of these sources is low, even at fainter magnitudes. However, multi-object spectrographs will help quantify the contaminating population and increases in throughput provided by modern IR detectors should help significantly. Ultimately, ground-based 30 m telescopes or space based spectroscopy from HST or JWST will be required to make significant progress towards spectroscopically confirming this population of objects.

Alternatively, a very promising means of confirming the redshift is through the [C ii] 158 μm line. This line is the dominant gas cooling line in the interstellar medium, tracing the cold neutral medium and photon-dominated regions at the interface with active star-forming regions, and it is now being routinely detected at z > 4 using existing submillimeter telescopes (Maiolino et al. 2005, 2009; Iono et al. 2006). Walter et al. (2009) show that the [C ii] line is a critical tracer of star formation in the first galaxies. Bootstrapping from the empirical relationship between [C ii] luminosity and star formation rate based on existing high-z samples, we expect a peak line flux density of ∼1 to 3 mJy for these sources. Such a line could be detected with the Plateau de Bure interferometer at 240 GHz in about 10 hr (Maiolino et al. 2009). Furthermore, once operational in 2013, ALMA will allow for high-resolution imaging of the gas on sub-kpc scales in even short integrations.

6.2. Implications for Galaxy Studies of the z > 7 Universe

A change in the shape of the luminosity function at z > 6 is not unexpected. In CDM models the knee of the galaxy luminosity function is usually explained by feedback which truncates star formation in the highest mass halos. If feedback is inefficient in the first generation of galaxies, their luminosity function should flatten out to follow the power-law halo mass function at these redshifts. The galaxy number density measured in this paper is below the expected halo density (Cooray & Ouchi 2006; Bouwens et al. 2008; Ouchi et al. 2009; Oesch et al. 2010), so we cannot rule these objects out based on space density alone. The most likely feedback mechanisms are AGNs, supernova, and stellar winds. But, in the early universe these mechanisms are likely to be inefficient. AGN feedback does not become efficient until the central black hole grows to a large enough mass to provide significant energy output. Simulations suggest this takes ∼200–500 Myr, leaving plenty of time for bright galaxies to exist at z > 6 (Li et al. 2007). Furthermore, simulations have shown that supernova do not contribute enough energy to truncate star formation, and may actually enhance star formation in a gas rich environment (Wise & Abel 2008). Finally, gas in the early universe will have very low metallicity, making winds less efficient at removing material.

Several empirical studies also support the existence of bright starbursts at z > 7. Studies of the z ∼ 4–6 mass function point to a epoch of star formation at z > 6 (Stark et al. 2009; McLure et al. 2009), and few studies suggest even more massive objects may exist (Wiklind et al. 2008; Eyles et al. 2007), but there is significant disagreement on this latter point (see Dunlop et al. 2007; Chary et al. 2007). Furthermore, z ∼ 6 quasars also exhibit significant enrichment (Kurk et al. 2007) and high dust contents (Riechers et al. 2007) implying a significant earlier generation of stars was present.

If we take the density of M ≃ 10¹¹ M_☉ galaxies to be ∼1 × 10⁻⁶ dex⁻¹ Mpc⁻³ at z ∼ 6 (Stark et al. 2009; McLure et al. 2009), the continuous star formation rate required to form these objects between z ∼ 20 and z ∼ 6 is >130 M_☉ yr⁻¹, which implies an intrinsic J-band magnitude of J < 23.6 at z = 7, placing an upper limit on the density of these systems of ∼4 × 10⁻⁷ mag⁻¹ Mpc⁻³. Star formation in bursts would make these objects even more luminous, but also rarer because they are short lived. Assuming a minimum formation time of 100 Myr for a 10¹¹ M_☉ galaxy yields a lower limit on the space density of ∼5 × 10⁻⁸ mag⁻¹ Mpc⁻³. Our measured value is reasonably between these two extremes.

In addition, there is significant disagreement about the evolution of the bright end of the luminosity function at z > 4, so the scale of disagreement between our result and those at z ∼ 6 is unclear. Deep ACS surveys suggest the density of bright galaxies is dropping at z > 4 (Bouwens et al. 2008). However, these surveys cover a very small volume and hence lack significant constraints on the bright end of the luminosity function. Larger area ground-based surveys provide mixed conclusions with Sawicki & Thompson (2006) and Iwata et al. (2007) pointing to no evolution in the density of bright galaxies while Yoshida et al. (2006) and Capak (2008) find mild evolution. McLure et al. (2009) find strong evolution, however their results are based on photometric redshfits which can miss a significant number of bright objects at these redshifts (Capak et al. 2011). In any case, the continuum luminosity function of spectroscopically confirmed Lyα emitters shown in Figure 15 also places a lower limit on the luminosity function at z ∼ 5.7 which is higher than the estimate from many studies at z ∼ 6 as is also noted by Hu & Cowie (2006).

Zoom In Zoom Out Reset image size

While it is numerically possible to fit a luminosity function to our data at z ∼ 7, the results would be misleading given the present uncertainty in the data. In addition to the uncertainty in the redshift of our sources, other surveys are likely dominated by cosmic shot noise at the bright end due to their small volumes (Overzier et al. 2009). At z > 5 the luminous galaxy population is extremely clustered, with bias factors of ∼6–10 (Hildebrandt et al. 2009; Ouchi et al. 2004). This means the mode (most common) luminosity function probed by small area surveys is likely to systematically underestimate the density of luminosity galaxies because the massive halos which host these massive (bright) systems are unlikely to occur in the survey volume. To account for this uncertainty, we have set the upper limits in Figure 14 to the point where they would detect three galaxies in an attempt to account for the presently unknown galaxy clustering properties.

The key to any successful Lyman break selection criteria is differentiating between the Lyman break and the 4000 Å break at lower redshifts and separating these objects from stars. For a 2 < z < 4 Lyman Break Galaxy (LBG) selection the low-redshift contamination is from z < 0.5 where the galaxy population is well understood along with O, B, and A stars which are relatively rare (Steidel et al. 2003). For a 4 < z < 6 selection contamination is from 0.5 < z < 1.5 galaxies where the population is not as well understood and M stars which are much more common (Stanway et al. 2008a; Capak et al. 2004). Furthermore, spectroscopic confirmation of both the high and low-redshift sources found by a 4 < z < 6 LBG selection is more difficult. Since these selection effects are not well understood this becomes a likely source of the large discrepancies seen between z ∼ 5 and 6 galaxy studies (Stanway et al. 2008a; Bouwens et al. 2007; Iwata et al. 2007). For a z > 7 LBG selection contamination comes from 1.5 < z < 2.5 galaxies and L, T, and Y dwarfs, all of which are poorly understood and extremely difficult to spectroscopically confirm. This is mitigated somewhat by Spitzer IRAC and K_s-band data which place strong constraints on the SED shape (Stanway et al. 2008b; Labbé et al. 2006). However, the strength of these constraints is limited by the lack of a representative set of object templates at 1.5 < z < 2.5 and the large uncertainties on deep Spitzer photometry due to confusion and under-sampling.

Of our three candidates, we find that object 1 is likely, but not conclusively at z = 7.7, object 2 is at z ∼ 1.6, and object 3 is very likely at z = 6.95. However, the faint H-band flux of object 3 is puzzling, suggesting a strong 2175 Å dust absorption feature at z = 6.95 or a quasar-type spectra with strong Lyα and C iv emission at z = 8.43. The spectral features observed in these sources are weak but known galaxy SEDs fail to produce a z ≪ 7 solution for these objects, and no objects at brighter magnitudes exhibit a similar SED, so together the evidence is reasonably strong.

Whether or not these sources are at z > 7, the difficulty in confirming even these bright objects suggests that deep small area surveys with small numbers of filters provide poor constraints on the z > 7 galaxy population. If the presented galaxies are at z > 7, then galaxies likely follow the dark matter halo mass function without a cutoff at the high-mass end as is observed at low redshift. As a result pencil beam surveys will fail to provide an accurate census of the galaxy population due to a lack of volume which prevents them from finding the most luminous galaxies and cosmic variance which increases their error. If we assume these galaxies are contaminated by z ≪ 7 objects, the z-dropout selection function needs to be quantified by spectroscopy before color selections can be used. If high equivalent width line emitters are the main source of contamination, then current NIR spectroscopy instruments may be able to determine this. However, direct spectroscopic confirmation of the z > 7 galaxy population will not be possible until thirty meter telescopes, JWST, or possibly ALMA are available. Either way, medium deep, pan-chromatic observations with adjacent filters and deep i- and z-band exposures will provide the best constraints on the z > 7 population for the foreseeable future.

The authors recognize and acknowledge the very significant cultural role and reverence that the summit of Mauna Kea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain. Support for this work was provided by the Spitzer Science Center which is operated by the Jet Propulsion Laboratory (JPL), California Institute of Technology under NASA contract 1407, NASA through contract 1278386 issued by the JPL and NASA grant HST-GO-09822. This work is based in part on observations made with the Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory, California Institute of Technology under a contract with NASA. Support for this work was provided by NASA through an award issued by JPL/Caltech. This research has made use of the NASA/ IPAC Infrared Science Archive, which is operated by the Jet Propulsion Laboratory, California Institute of Technology, under contract with the National Aeronautics and Space Administration. H.J.M. and J.P.K. acknowledge support from the French Agene National de la Recheche fund ANR-07-BLAN-0228 as well as from CNES and the Programme National Cosmologie et Galaxies. P.C. acknowledges the Keck remote observing staff who allowed him to simultaneously attend the NIRSPEC observations and the birth of his daughter.