The Rest-frame Optical Sizes of Massive Galaxies with Suppressed Star Formation at z ∼ 4

Mariko Kubo; Masayuki Tanaka; Kiyoto Yabe; Sune Toft; Mikkel Stockmann; Carlos Gómez-Guijarro

doi:10.3847/1538-4357/aae3e8

1. Introduction

There is mounting evidence for the presence of massive galaxies with suppressed star formation at z > 2 (e.g., Daddi et al. 2005; van Dokkum et al. 2008). These galaxies are known to be remarkably compact and dense compared to local ones (e.g., Trujillo et al. 2006; Toft et al. 2007; van Dokkum et al. 2008; Buitrago et al. 2008; van der Wel et al. 2014; Kubo et al. 2017). The size evolution of these massive quiescent galaxies (QGs) can be parameterized as $r\propto {(1+z)}^{\beta }$ , where β ∼ −1.5, which is steeper than the β ∼ −1 of star-forming galaxies (SFGs; e.g., van der Wel et al. 2014; Shibuya et al. 2015).

The remarkable compactness and early formation of massive QGs pose a challenge to the standard picture of galaxy formation in which galaxies grow hierarchically and become more massive with time. Gas-rich major mergers (e.g., Hopkins et al. 2008; Wellons et al. 2015) and infall of giant clumps formed via disk instability (e.g., Elmegreen et al. 2008; Dekel et al. 2009) can trigger nuclear starburst and increase the central density in galaxies to form a compact remnant. Discoveries of compact starburst galaxies at z > 2 may support these scenarios (Barro et al. 2014; Toft et al. 2014; Ikarashi et al. 2015; Barro et al. 2016, 2017; Ikarashi et al. 2017; Gómez-Guijarro et al. 2018). On the other hand, massive QGs at high redshift need several to 10 times the growth in size but less growth in stellar mass to evolve into giant elliptical galaxies today. Dry minor mergers (e.g., Bezanson et al. 2009; Naab et al. 2009) and the adiabatic expansion (Fan et al. 2008; van Dokkum et al. 2014) and size evolution of newly quenched galaxies with redshift (Carollo et al. 2013; Poggianti et al. 2013; Belli et al. 2015) have been proposed as the driver of this steep size growth.

Now massive QGs at z ∼ 4 are found photometrically (Straatman et al. 2014) and confirmed spectroscopically (z_spec = 3.717; Glazebrook et al. 2017; Schreiber et al. 2018). The Hubble Space Telescope (HST) has been the main workhorse in the field of galaxy morphologies at high redshift, but it cannot probe the rest-frame optical wavelength regime of galaxies at z > 3 due to its wavelength cutoff of ∼1.7 μm. In this study, we select galaxies with a prominent Balmer break feature at z ∼ 4 photometrically from the Subaru XMM-Newton Deep Survey (SXDF; Furusawa et al. 2008) and investigate their rest-frame optical morphologies by the deep K'-band images obtained with the adaptive optics (AO) on the Subaru Telescope.

This paper is organized as follows: in Section 2 we describe our sample selection of massive galaxies with suppressed star formation, in Section 3 we describe the observation and data reduction procedure, in Section 4 we describe the size measurement method and possible errors, and in Section 5 we show the results. We discuss the stellar mass surface density and their size–stellar mass evolution in Section 6. Throughout the paper, we adopt a ΛCDM cosmology with H₀ = 70 km s⁻¹ Mpc⁻¹, Ω_Λ = 0.7, and Ω_m = 0.3, and magnitudes are given in the AB system.

2. Sample Construction

2.1. Multiband Catalog

We based our analysis on a multiband photometric catalog in the Subaru XMM-Newton Deep Field (SXDF; Furusawa et al. 2008). SXDF has deep optical imaging from the Suprime-Cam of the Subaru Telescope in the BVRiz-bands (Furusawa et al. 2008). The UKIRT Infrared Deep Sky Survey (UKIDSS; Lawrence et al. 2007) is centered on the same field, and we used Data Release 10 to complement the optical data. Furthermore, the u-band photometry from CFHT Megacam and Spitzer photometry from the Spitzer UKIDSS Ultra Deep Survey (Dunlop et al. 2007) are available, allowing us to cover the entire optical and IR wavelengths up to 24 μm over a wide area. It is an excellent field to search for faint, rare objects at high redshifts.

We first registered all of the optical images to the WCS grid of the UKIDSS images. The seeing is different from band to band, and we applied a Gaussian kernel to homogenize the seeing to ∼0.82 arcsec. We ran SExtractor (Bertin & Arnouts 1996) on the K-band image to detect sources. We then measured sources in the other optical and near-IR bands using the dual image mode. We performed photometry within a circular aperture of 2.0 arcsec in all the bands. Because we missed a fraction of total light in this aperture, we measured the Kron fluxes of objects in the K-band and estimated the aperture correction, assuming that the Kron flux is the total flux (hereafter we refer to the Kron magnitude as the total magnitude). We applied the aperture correction to the 2.0 arcsec aperture photometry in all of the bands so that our photometry is closer to total light while keeping accurate colors.

Because of the relatively large PSF sizes of the Spitzer/IRAC images, objects are often blended with nearby objects, and we chose to perform the Spitzer photometry separately from the optical–near-IR bands. We use T-PHOT (Merlin et al. 2016) version 1.5.11 to fit 2D profiles of objects in the IRAC images, taking the object blending into account using the K-band image as a prior. For objects detected in the K-band high-resolution image (HRI), small image cutouts of the same region were generated in order to model the IRAC low-resolution image (LRI). The cutouts were convolved with a kernel constructed from LRI and HRI, both of which were constructed from point sources selected in HRI, to homogenize the PSF. Then, the optimization process was performed by scaling the fluxes of the objects of the PSF-matched HRI to match the LRI using the χ² minimization technique. We processed the IRAC images in all channels from 3.6 to 8.0 μm in the same way, and we used the total magnitude of each object from the best-fit model flux.

In the final catalog, we have about 10⁵ objects over ∼0.7 deg² with coverage in all of the filters. Table 1 summarizes the depth in each band.

Table 1. 5σ Limiting Magnitudes Within 2''Apertures for Each Filter

Filter	Instrument	Depth
u	Megacam	26.8
B	Suprime-Cam	27.6
V	Suprime-Cam	27.3
R	Suprime-Cam	27.1
i	Suprime-Cam	27.0
z	Suprime-Cam	26.0
J	WFCAM	25.2
H	WFCAM	24.6
K	WFCAM	25.0
ch1	IRAC	24.8
ch2	IRAC	24.3
ch3	IRAC	22.6
ch4	IRAC	22.5

Download table as: ASCII Typeset image

2.2. Target Selection

We ran a custom photometric redshift code (Tanaka 2015) on the multiband catalog. This is a template-fitting code, and we used templates generated using the Bruzual & Charlot (2003) stellar population synthesis code. We adopted the following assumptions in the models: exponentially declining star formation history, solar metallicity, Calzetti et al. (1994) attenuation curve, and Chabrier (2003) initial mass function (IMF). As we know the SFR and attenuation of each template, we added emission lines due to star formation using the emission line intensity ratios of Inoue (2011; see Tanaka 2015 for details). The code infers the redshifts and physical properties of galaxies such as stellar mass in a self-consistent manner, and the uncertainties on the physical properties quoted in the paper have been estimated by marginalizing over all the other parameters, including redshift. As we have a large number of filters spanning a wide wavelength range, the data have a strong constraining power on the overall SED shapes. We therefore chose to apply flat priors in the fitting. We have confirmed that our results do not significantly change if we apply the full priors. Using some of the publicly available spectroscopic redshifts (Bradshaw et al. 2013; McLure et al. 2013; J. M. Simpson et al. 2018, in preparation), we achieved a normalized dispersion of σ(Δ z/(1 + z)) = 0.029 and an outlier rate of 4.8%, where the outliers are defined in the conventional way (i.e., those with $| {\rm{\Delta }}z/(1+z)| \,\gt 0.15;$ Tanaka et al. 2018). However, the spectroscopic sample is heterogeneous, and the numbers here should not be overinterpreted.

We excluded objects with unreliable photo-z's using the reduced chi-squares, χ_ν > 4. Poor chi-squares are often due to poor photometry (e.g., halos around bright stars and object blending). For the purpose of this paper, we do not need a complete sample of evolved galaxies at high redshift, and this cut does not introduce any bias. We then selected galaxies at 3.5 < z_phot < 4.5. Figure 1 shows the star formation rate (SFR) against stellar mass of the z ∼ 4 galaxies. Both SFR and stellar mass are from the SED fit. There is a clear sequence of SFGs and also a population of massive galaxies with suppressed star formation. These two populations can be separated very well at a specific SFR (sSFR) of 10^−9.5 yr⁻¹. To be conservative, we chose galaxies with sSFR 1σ upper limit lower than 10^−9.5 yr⁻¹ as the targets for the near-IR follow-up imaging with AO. The red filled points in Figure 1 satisfy this condition. We note that there is some ambiguity in the definition of QGs in the literature, but when we refer to QGs in what follows, we mean galaxies with suppressed star formation as defined in Figure 1. The UVJ diagram is often used to define QGs, but it is tuned at z ≲ 2 (Labbé et al. 2005; Williams et al. 2009) and is not clear whether it can be applied to z ∼ 4 galaxies. For this reason, we adopt the sSFR-based definition.

**Figure 1.** SFR vs. stellar mass of galaxies at z ∼ 4. The unfilled circles are SFGs. The filled circles are QG candidates with the 1σ upper limit of the sSFR lower than 10^−9.5 yr⁻¹. Objects with SFRs smaller than 1 M_⊙ yr⁻¹ are shown at SFR = 1 M_⊙ yr⁻¹ only for illustrative purposes. The stars indicate the targets observed with IRCS+AO188 (see Section 3). The unfilled star is ID3, the target also classified as quiescent by Straatman et al. (2014).
Download figure:
Standard image High-resolution image

In addition to the sSFR constraint, further practical constraints come from the location of tip-tilt stars for the AO-assisted observation. Since we need tip-tilt stars R = 16.5 or brighter (for NGS mode, R < 19 for LGS mode) for AO188, the available targets are further limited. We have conducted near-IR follow-up imaging with AO for five of the brightest QGs with suitable tip-tilt stars as shown by the stars in Figure 1 (hereafter ID1-5). Figure 2 shows their SEDs. All of them are located around z_phot ∼ 4. As can be seen, all of the objects show a prominent Balmer break, indicative of an evolved stellar population. ID1 and ID2 have a faint UV continuum and are consistent with passively evolving galaxies (their SED-based SFR is consistent with zero). The others have a brighter UV continuum, but the break feature is still prominent. To further characterize our targets, we compare the mean SEDs of SFGs with that of QGs in Figure 3. SFGs have a very blue UV continuum with a strong Lyman break. On the other hand, the SEDs of our targets are clearly distinct; they have a suppressed UV continuum with a clear Balmer break. This break cannot be due to dust extinction because it does not introduce a sudden break at 3650 Å while keeping the continuum at longer wavelengths blue. This is due to abundant A-type stars in these galaxies. The observed targets are consistent with the mean SED shown by the red shades and suggests that they are representative of the evolved population around that redshift.

**Figure 2.** Top: the SEDs of our targets, ID1 to ID5. The spectrum is the best-fitting template spectrum and the points are the observed photometry. Some of the relevant quantities such as age and star formation timescale of the template are also indicated. Bottom: redshift probability distribution function. The arrow shows the median redshift.
Download figure:
Standard image High-resolution image

**Figure 3.** Rest-frame mean SEDs of SFGs (blue) and QGs (red) normalized in the V-band. The shaded areas encompass the 68th percentile of the distribution. The objects that we observed are shown as the solid lines. They show the prominent Balmer break.
Download figure:
Standard image High-resolution image

We note that a part of our survey area is observed in the Fourstar Galaxy Evolution survey (ZFOURGE; Straatman et al. 2016). Straatman et al. (2014) selected QGs at z ∼ 4 using the rest-frame UVJ colors and photometric redshifts from ZFOURGE. We briefly compare our sample of QGs with those in Straatman et al. (2014). We find that the QGs identified in SXDF (UDS) in Straatman et al. (2014) all satisfy sSFR < 10^−9.5 yr⁻¹ based on our catalog. On the other hand, two of our targets, ID3 and ID5, are in the ZFOURGE field. ID3 is also identified as a QG in ZFOURGE, whereas ID5 is not. The rest-frame color of ID5 is U − V = 0.95 ± 0.04 and V − J = 0.86 ± 0.02 (Straatman et al. 2016), slightly bluer than the color criterion for QGs adopted in Straatman et al. (2014), but their SED fit suggests sSFR < 10^−9.5 yr⁻¹ at z_phot ≈ 4, satisfying our criterion of QGs. Overall, our QG selection is broadly compatible with that of Straatman et al. (2014). It is noteworthy that most of the QGs in their sample are fainter than K > 23. Thanks to the wider area coverage, most of our targets are brighter and better suited for detailed structural studies.

3. Observation and Data Reduction

We observed the five targets selected in Section 2 with IRCS (Tokunaga et al. 1998; Kobayashi et al. 2000)+AO188 (Hayano et al. 2008, 2010) on the Subaru Telescope on 2016 September 25 and 26. We used the K'-filter with the 52 mas pixel scale. The observing conditions were fair; the sky was clear on both nights with reasonably good seeing (∼0.2 arcsec with AO), though it fluctuated occasionally. We observed in both NGS and LGS modes due to occasional poor seeing and satellite crossings. We rejected the worst ∼10% of the bad seeing frames. After rejecting these bad PSF frames, the variation of the PSF sizes of the frames on each target is less than 0.05 arcsec.

We reduced the data using the IRAF data reduction tasks following the data reduction manual for the IRCS.⁵ We first masked bad pixels and then applied the flat, which were constructed from dithered science exposures with objects masked out. The sky background is the median value in the whole area of each frame, ∼54 arcsec on a side. We estimated the telescope offset between the pointing from the relative positions of bright stars within the field of view. Finally, we combined the frames with 3σ clipping.

Magnitude zero points were calibrated by using the K-band images of UKIDSS. We estimated the K − K' (i.e., WFCAM − IRCS) color as a function of J − K color using the stellar library from Pickles (1998). We set the zero points of the IRCS-AO K'-band images by matching the fluxes of bright (K < 21) but not saturated stars with those measured on the UKIDSS K-band images after applying the $K-K^{\prime}$ color term. The $K-K^{\prime}$ colors of the stars used as the standard stars here range from −0.12 to −0.10. Since observing conditions were stable during the nights, we used the average of the magnitude zero points of each night, 25.41 for September 25 (ID2 and 3) and 25.43 for September 26 (ID1, 4, and 5).

We summarize the details of the coadd images in Table 2. The total exposure time of each target ranges from 18 to 54 minutes. The FWHM PSF sizes measured on the PSF reference stars range from 0.15 to 0.23 arcsec.

Table 2. Summary of Observations

ID	R.A.	Decl.	EXPTIME	ZEROPOINT	Depth^a	Separation^b	FWHM PSF^c
	(h:m:s)	(d:m:s)	(min)	(mag)	(mag)	(arcsec)	(arcsec)
1	02:19:01.511	−05:18:29.07	33	25.43	24.7	72(33)	0.17
2	02:17:59.073	−05:09:39.89	18	25.43	24.6	53(34)	0.21
3	02:17:22.781	−05:17:33.34	35	25.41	24.9	48(16)	0.15
4	02:17:19.833	−04:43:34.75	43	25.43	25.0	41(38)	0.23
5	02:16:58.232	−05:08:35.21	54	25.41	25.0	37(13)	0.19

Notes.

^a5σ limiting magnitudes measured with 0.3 arcsec diameter aperture. ^bThe separation between the tip-tilt stars and the targets. The numbers in parentheses are the separations between the tip-tilt stars and the PSF reference stars. ^cFWHM of the PSF reference stars.

Download table as: ASCII Typeset image

4. Size Measurement

4.1. Flux Completeness

We first examined the flux completeness of our targets on the IRCS-AO K'-band images by comparing the flux measured on the IRCS-AO K'-band and UKIDSS K-band images. The signal-to-noise ratios (S/Ns) on our K'-band images are lower than that on the UKIDSS K-band images. Then, if our targets have morphologies dominated by low-surface brightness components, a large fraction of their fluxes detectable on the UKIDSS K-band images may go below the detection limit on our IRCS-AO K'-band images. Also, the AO-corrected PSF tends to have an extended wing, which also introduces a diffuse component in the observed profiles. These effects can result in underestimated sizes and fluxes.

We compared the K'-band total magnitudes measured on our IRCS-AO K'-band ( ${K}_{\mathrm{total},\mathrm{IRCS}-\mathrm{AO}}^{{\prime} }$ ) with the UKIDSS K-band magnitudes corrected with the $K-K^{\prime}$ color term using the best-fit SED model ( ${K}_{\mathrm{total},\mathrm{synth}}^{{\prime} }$ ) in order to evaluate the missing flux (Figure 4 and Table 3). Overall, we tended to underestimate the fluxes in the IRCS-AO images as expected. For ID4 and ID5, we underestimated only by 10%, and the missing light probably did not affect our size measurements significantly. However, we missed 25%–50% of the light for the other targets. Though care is needed when interpreting individual galaxies, the stacked galaxy (unfilled circle; see Section 4.4) does not show a significant amount of missing flux, suggesting that its size can be robustly measured. We make an attempt to estimate the effects of the missing light on the size measurements in Section 4.3, where we actually reproduce the amount of missing fluxes with a simulation and evaluate the limitation from the PSF.

**Figure 4.** Synthetic K'-band magnitude ( ${K}_{\mathrm{total},\mathrm{synth}}^{{\prime} }$ ) plotted against the IRCS K'-band magnitude ( ${K}_{\mathrm{total},\mathrm{IRCS}-\mathrm{AO}}^{{\prime} }$ ) of our targets. The red filled circles show the individual objects. The red unfilled circle shows the stack of ID1 to ID4.
Download figure:
Standard image High-resolution image

**Figure 4.** Synthetic K'-band magnitude ( ${K}_{\mathrm{total},\mathrm{synth}}^{{\prime} }$ ) plotted against the IRCS K'-band magnitude ( ${K}_{\mathrm{total},\mathrm{IRCS}-\mathrm{AO}}^{{\prime} }$ ) of our targets. The red filled circles show the individual objects. The red unfilled circle shows the stack of ID1 to ID4.
Download figure:
Standard image High-resolution image

Table 3. Properties of the Observed Objects

ID	z_phot	K_tot^a	${K}_{\mathrm{tot},\mathrm{synth}}^{{\prime} }$ ^b	${K}_{\mathrm{tot},\mathrm{observed}}^{{\prime} }$ ^c	r_e,maj^d	M_⋆
		(mag)	(mag)	(mag)	(kpc)	( ${10}^{11}\,{M}_{\odot }$ )
1	4.07	22.47 ± 0.05	${22.60}_{-0.05}^{+0.05}$	23.26 ± 0.06	0.92 ± 0.31	1.58
2	3.83	22.54 ± 0.05	${22.69}_{-0.05}^{+0.06}$	23.43 ± 0.08	0.22 ± 0.21	1.09
3	3.70	22.55 ± 0.05	${22.65}_{-0.05}^{+0.05}$	23.07 ± 0.04	0.63 ± 0.18	1.04
4	4.24	22.61 ± 0.05	${22.73}_{-0.05}^{+0.05}$	22.85 ± 0.03	0.50 ± 0.21	1.83
5	4.09	23.35 ± 0.09	${23.38}_{-0.04}^{+0.10}$	23.52 ± 0.05	1.70 ± 0.71	1.13
STACK	⋯	22.54 ± 0.03	${22.67}_{-0.03}^{+0.03}$	22.78 ± 0.03	0.52 ± 0.18	1.38

Notes.

^aKron magnitudes measured on the UKIDSS K-band images. ^bExpected K'-band total magnitudes from the SED fits. ^cKron magnitudes measured on the IRCS-AO K'-band images. ^dMedian and standard deviation of the r_e,maj measured with fixed n = 0.5, 1, 2, 3, 4, and 5.

Download table as: ASCII Typeset image

4.2. GALFIT Fitting

The sizes of our targets were measured by fitting Sérsic profiles (Sersic 1968) to their K'-band images using GALFIT (Peng et al. 2002, 2010). GALFIT fits two-dimensional analytical functions convolved with a PSF to an observed galaxy image. Here, we used a scale of 6.951 kpc arcsec⁻¹ in the physical at z = 4 for all the targets. We used the nearest star in the field of view or a star taken before and after the science exposures as the PSF reference star. We at first fit the Sérsic models in ranges of effective radius r_e = 0.2–12 kpc and Sérsic index n = 0.5–10. The fits were performed using an image cutout of 3.0 arcsec on a side for each object. The background values were estimated in an annulus between 2.9 and 3.0 arcsec from each object before Sérsic model fittings. As an initial guess, we used total magnitudes measured by SExtractor (Bertin & Arnouts 1996), r_e = 1 kpc and n = 1.4. The results are not sensitive to this initial guess. In order to compare our results with those of van der Wel et al. (2014), who measured galaxy sizes out to z ∼ 3, we here used the effective radius along the semimajor axis (r_e,maj).

4.3. GALFIT Fitting Errors

In Kubo et al. (2017), the morphologies of galaxies at z ≈ 3 were studied by using the deeper K'-band image taken with the same instrument as ours and discussed the errors of the GALFIT fittings on those images. We here discuss the possible errors in our size measurements following that work.

Let us start with the limitation from the PSF. We are now studying the targets that can be hardly resolved even with our high-resolution images. We should note that our results can be just an upper limit since the reduced χ² values of the Sérsic model fitting only marginally (Δχ² ∼ 0.01) improves over that with PSF model fitting. In addition, the fits with models of different Sérsic indices n = 0.5 ∼ 5 are equally good. Then, we adopted the median r_e,maj from the GALFIT fitting with n = 0.5, 1, 2, 3, 4, and 5 as the best-fit values.

In addition, there can be errors originating from a little PSF inconsistency. We ideally need to evaluate the PSF at the positions of the targets, but that is in practice difficult. We use a single PSF reference star either within the field of view or taken before/after the science exposures. Even though the target and PSF reference stars are taken in the same frame, as shown in Table 2, the distance between the tip-tilt star and the target, and that between the tip-tilt star and the PSF reference star is not the same. In the case of our targets, we expect the PSF difference of ≲0.03 arcsec according to the performance of AO188.⁶ However, since the size of galaxies at z ∼ 4 is very small, this may not be negligible. The separation between the PSF reference star and the tip-tilt star is always smaller than the separation between the target and the tip-tilt star, i.e., the PSF we use in the fits is likely smaller than the real PSF at the object position. This leads us to overestimate the size. Thus, our estimates are likely conservative. Kubo et al. (2017) reported that this level of PSF inconsistency does not affect the measured sizes, but on the other hand, it significantly affects the measured Sérsic indices, which we do not discuss in this paper.

Next, we tested the accuracy of the GALFIT measurement by generating mock galaxy images following Kubo et al. (2017). We investigated the typical fitting errors by inserting artificial objects on the sky of the observed image, measuring their sizes and comparing the input and output structural parameters. We here used the coadd image of ID1 as the representative case of our sample. We generated artificial sources over a range of parameters: $K^{\prime} =22.6$ ( $\approx {K}_{\mathrm{tot},\mathrm{synth}}^{{\prime} }$ of ID1), r_e,maj < 4 kpc, Sérsic indices n = 0–8, and various axis ratios and position angles. They are convolved with the PSF reference star for ID1 and added to the sky of the ID1 image. By repeating this simulation, we found that the median and standard deviation of the measured total magnitude are 23.0 ± 0.6 in the case ${r}_{e,\mathrm{maj},\mathrm{model}}=0.5\sim 2\,\mathrm{kpc}$ , which is consistent with the observed K'-band total magnitude of ID1. In other words, the simulation reproduces the missing flux in the observation, suggesting that our simulation is reasonably realistic. We show the r_e,maj of the mock galaxies ( ${r}_{e,\mathrm{maj},\mathrm{model}}$ ) versus those measured with GALFIT ( ${r}_{e,\mathrm{maj},\mathrm{model}}$ ) in Figure 5. Naively, we expect that the sizes of small (≲1 kpc) objects are overestimated from the input sizes due to the limited resolution while large objects are underestimated since their outer profiles are buried in noise. We can see this tendency weakly. The standard deviation of Sérsic indices is σ(n) = 2.3 (plot not shown). This again suggests that the Sérsic indices are hardly constrained by our data. As for axis ratio, we can obtain a reasonable estimate when galaxies are highly elongated (b/a ≲ 0.3), but otherwise it is essentially unconstrained. This may be caused by the asymmetric distribution of noisy pixels around the source, since this tendency is softened at the depth of the stacked image. The best-fit models of ID1 and ID5 in Figure 6 look elongated; however, it is not clear that they are real signatures.

**Figure 5.** Top: sizes measured by GALFIT ( ${r}_{e,\mathrm{maj},\mathrm{fit}}$ ) as a function of input size ( ${r}_{e,\mathrm{maj},\mathrm{model}}$ ) in our simulation for ID1. The solid line shows ${r}_{e,\mathrm{maj},\mathrm{model}}={r}_{e,\mathrm{maj},\mathrm{fit}}$ . The dashed curve and gray shaded regions show the median and 1σ range of the ${r}_{e,\mathrm{maj},\mathrm{fit}}$ at the given ${r}_{e,\mathrm{maj},\mathrm{model}}$ . Bottom: similar to the top panel but for the stacked galaxy.
Download figure:
Standard image High-resolution image

**Figure 5.** Top: sizes measured by GALFIT ( ${r}_{e,\mathrm{maj},\mathrm{fit}}$ ) as a function of input size ( ${r}_{e,\mathrm{maj},\mathrm{model}}$ ) in our simulation for ID1. The solid line shows ${r}_{e,\mathrm{maj},\mathrm{model}}={r}_{e,\mathrm{maj},\mathrm{fit}}$ . The dashed curve and gray shaded regions show the median and 1σ range of the ${r}_{e,\mathrm{maj},\mathrm{fit}}$ at the given ${r}_{e,\mathrm{maj},\mathrm{model}}$ . Bottom: similar to the top panel but for the stacked galaxy.
Download figure:
Standard image High-resolution image

**Figure 6.** Observed images and GALFIT result for ID1 to ID5 and the stacked galaxy. The observed images, best-fit Sérsic models, and residuals are shown from left to right. The sizes of the images are 2.1 arcsec × 2.1 arcsec. The yellow filled circles show the FWHM of the PSF size on each image.
Download figure:
Standard image High-resolution image

Finally, we compared the r_e,maj measured from the HST/WFC3 H-band image and our K'-band images. Among our sample, only ID5 is within the Cosmic Assembly Near-IR Deep Extragalactic Legacy Survey (CANDELS; Grogin et al. 2011; Koekemoer et al. 2011). We summarize the comparison in Table. 4. They are not necessarily the same as our results due to the wavelength difference, but they are a good comparison. The size estimates on these images are broadly consistent with each other, but our size estimate is slightly larger as expected from the simulation above. This may also imply that they show no strong rest-frame UV to optical color gradient due to the age and/or metallicity gradient of the stellar population as well as attenuation by dust. The stellar population of these galaxies may be relatively simple. On the other hand, the uncertainty in the Sérsic indices is large for K', which is again consistent with the above indications.

Table 4. GALFIT Fittings of ID5 with the IRCS-AO K'-band and WFC3 H-band Images

Band	mag	r_e,maj	n
	(mag)	(kpc)
K'	23.52 ± 0.05	1.70 ± 0.71	${0.79}_{-0.5}^{+2.99}$
H	24.84 ± 0.02	1.19 ± 0.03	0.73 ± 0.03

Download table as: ASCII Typeset image

Taking all the tests together, there is a small bias in our size measurements for individual objects in the sense that we tend to overestimate the sizes by ∼30%. We did not account for this bias just to be conservative. Our estimates can thus be considered as reasonable upper limits.

4.4. Stacking Analysis

We stacked our targets to gain S/N and measured their average size. We excluded ID5 from the stacking because it is fainter than the others. We smoothed the single-exposure images of ID1 to ID4 to a common seeing of 0.23 arcsec (that of ID4) by convolving with a Gaussian, and then performed median stacking of these images. The total K'-band magnitude measured on the stacked image shows only a small amount of missing flux (10%; Figure 4).

We repeated the same GALFIT simulation using the stacked image and PSF reference star for ID4 (Figure 5, bottom). Similar to the individual galaxies, the reduced χ² values of the Sérsic model fitting only marginally improves from that of the PSF model fitting. The r_e,maj errors are reduced greatly from the simulation for ID1. The bias in the size measurement marginally changes depending on the PSF adopted. This gives us confidence in the measured sizes of the QGs on our stacked image.

5. Results

The results of GALFIT fitting are shown in Figure 6 and summarized in Table 3. The r_e,maj of our targets range from 0.2–1.7 kpc with the median and standard deviation being 0.6 kpc and 0.6 kpc, respectively. Our results indicate that massive QGs at z ∼ 4 are indeed compact. As discussed above, the individual size estimates may suffer from flux incompleteness (we are missing diffuse light), but we obtain a consistent result for the stacked galaxy; the r_e,maj measured on the stack is 0.52 ± 0.18 kpc, providing further support for the compact sizes. We also note that the possible systematic errors from the PSF inconsistency are not included in our size estimate errors; however, as we mentioned above, it may not affect them significantly.

Figure 7 shows the stellar mass versus r_e,maj diagram of the QGs at z ∼ 4. For comparison, we plot the size–stellar mass relation of QGs at z = 2.75 measured at the rest-frame optical (van der Wel et al. 2014) and at z ∼ 3.7 measured at the rest-frame UV (Straatman et al. 2015 using the catalog in Straatman et al. 2014 described above). Both studies selected QGs with photometric redshifts and UVJ colors, and measured the size on HST/WFC3 H-band images from CANDELS. The QGs at z ∼ 4 are below the size–stellar mass relation of QGs at z = 2.75, suggesting that they have physical sizes smaller than those of lower redshift ones. The size measured on the stack shown with the unfilled square confirms this trend. The QGs at z ∼ 3.7 have a somewhat large dispersion in size, but our targets have sizes consistent with some of their smallest objects. There are a few objects with the large size of r_e,maj ∼ 4 kpc among QGs at z ∼ 3.7, which is more consistent with typical sizes of SFGs at z = 2.75 (van der Wel et al. 2014). This might indicate the contamination of SFGs in their UVJ-selected QGs.

In Figure 8, we show the rest-frame optical size–redshift relation of galaxies with 10¹¹ M_⊙ ≤ M_⋆ ≤ 10^11.5 M_⊙. The r_e,maj at z = 0, 0.75 ≤ z ≤ 2.75, and z ∼ 3.7 are the median sizes of QGs with 10¹¹ M_⊙ ≤ M_⋆ ≤ 10^11.5 M_⊙ from Guo et al. (2009), van der Wel et al. (2014), and Straatman et al. (2015), respectively. The z = 3.1 point is from Kubo et al. (2017), who measured the size of a QG with M_⋆ ≈ 2 × 10¹¹ M_⊙ in a protocluster at the K'-band using IRCS/AO188. Our stacked galaxy is shown by the unfilled square. We extend the size–redshift relation of QGs out to z ∼ 4 for the first time. The figure shows that the sizes of massive QGs continuously decrease with redshift up to z = 4, an order of magnitude size evolution between z = 4 and 0. This is a surprisingly strong evolution. Note that the size–stellar mass relation in van der Wel et al. (2014) shows an upturn at z = 2.75. This could be caused by the contamination of SFGs. In their UVJ color diagram, the dispersions of the color sequences of QGs and SFGs increase with redshift due to the observational errors and maybe the change of the SEDs of the galaxies. Then it is expected that contaminants in galaxies selected as QGs increase with redshift. Straatman et al. (2015) also used rest-frame UVJ color selection, but since they did not use the sample near the border of selection criterion, such contaminants may be reduced in their sample.

**Figure 8.** Size evolution of QGs with stellar mass 10¹¹ M_⊙ ≤ M_⋆ ≤ 10^11.5 M_⊙ up to z = 4. The blue unfilled square shows the stack of QGs at z ∼ 4 in this study. The red unfilled circle, filled circles and cross show the median r_e,maj of QGs at z = 0 from Guo et al. (2009), at 0.75 ≤ z ≤ 2.75 from van der Wel et al. (2014), and at z ∼ 3.7 from Straatman et al. (2015), respectively. The error bar of Guo et al. (2009) shows the ∼0.1 dex difference between Shen et al. (2003) and Guo et al. (2009). The blue unfilled triangle shows the QG with M_⋆ ≈ 2 × 10¹¹ M_⊙ at z = 3.1 in Kubo et al. (2017). The black solid curve shows the ${r}_{e,\mathrm{maj}}=A{(1+z)}^{\beta }$ fit in this study. The gray dashed and dotted lines show those for QGs and SFGs in van der Wel et al. (2014), respectively.
Download figure:
Standard image High-resolution image

The size–redshift relation is often parameterized in the form r_e kpc⁻¹ = A(1 + z)^β. van der Wel et al. (2014) found $A\,={11.2}_{-2.1}^{+2.6}$ and β = −1.32 ± 0.21 for QGs with 10¹¹ M_⊙ ≤ M_⋆ ≤ 10^11.5 M_⊙ (dashed line in Figure 8). Adding the results at z > 3 and fitting at 0.75 ≤ z ≤ 4, we find A = 18.8 ± 3.0 and β = −1.9 ± 0.2 (solid line), though it is hard to fit the entire redshift range with this form. Our results support a stronger size evolution of QGs compared to SFGs with β ∼ −1 (e.g., van der Wel et al. 2014; Shibuya et al. 2015; Straatman et al. 2015) up to z = 4.

6. Discussion

In this study, we measured the size of massive QGs at z ∼ 4 in the rest-frame optical wavelength for the first time based on the AO-assisted imaging using a ground-based telescope. There are a few possible uncertainties in our results.

One is the contamination from active galactic nuclei (AGNs), which could make galaxies look compact. However, as shown in Figure 2, the overall SEDs of our targets are dominated by evolved stellar populations as indicated by the strong Balmer break, which suggests that the continuum is dominated by stars. Thus, the AGN contamination, if any, is unlikely to significantly alter our results. Our targets are not detected in X-rays (Ueda et al. 2008) or MIR (Dunlop et al. 2007). Although only very active AGNs are detectable at the depth of the data at z ∼ 4, this adds further support for no significant AGN contamination.

There is another question regarding the quiescence of our targets. Although the SED fits indicate that these galaxies are not actively forming stars, their quiescence should be further confirmed by other means. Gobat et al. (2018) detected significant far-IR fluxes from BzK- and UVJ-selected QGs, suggesting that the optical–near-IR selection does not always give a clean sample of QGs. Multiwavelength follow-up observations of our targets are essential to fully confirm their quiescence. Efforts in this direction are underway.

Although we should further address these possible uncertainties in the future, it is interesting to discuss the origin and evolution of these extremely compact massive QGs at z ∼ 4. In this section, we first discuss their extremely high stellar mass surface density and then focus on their size evolution on the evolving stellar mass track.

6.1. Extremely High Stellar Mass Surface Density

It has been known that massive QGs at high redshift have extremely high stellar mass surface densities (e.g., van Dokkum et al. 2008). We compare the mean stellar mass surface densities within the effective radii of massive QGs at z = 4 and dispersion-supported stellar systems in the local universe (Brodie et al. 2011) in Figure 9. Brodie et al. (2011) was originally given in V-band luminosity. We converted the V-band luminosity into stellar mass by adopting M_⋆/L_V = 3, which is for the case of a simple stellar population model with the age of ∼10 Gyr, adopting the Chabrier (2003) IMF. Note that M_⋆/L_V can depend on the object type. We also show the densest ultracompact dwarf reported in Strader et al. (2013) using its stellar mass from Seth et al. (2014).

**Figure 9.** Surface stellar mass densities within effective radii versus stellar mass. The large blue diamond shows massive QGs at z = 4. We also show its evolution track found in Section 6.2. The crosses show dispersion-supported systems in the local universe from Brodie et al. (2011; GC = globular cluster; cE = compact elliptical; E = early type galaxy; dE = dwarf elliptical). The red unfilled circle shows the densest UCD reported in Strader et al. (2013).
Download figure:
Standard image High-resolution image

It is interesting that high-z QGs and globular clusters (GCs), consisting of the oldest stars of the Milky Way and thought to form at high redshifts, both have extremely high stellar mass surface densities, even though their typical masses differ by several orders of magnitude. Johnson et al. (2017) and Vanzella et al. (2017) found very low mass (M_⋆ = a few 10⁶ M_⊙), extremely dense galaxies at z = 2–6 with strong lensing. Though they are more massive than GCs, they imply that such ultradense objects are commonly formed at high redshift. Given the high density and high gas fraction in the early universe, we naturally expect that gas-rich major mergers are one of the channels for forming such extremely compact objects. In addition, cosmological numerical simulations predict that high-z galaxies are fed by streams of smooth gas and merging clumps from the cosmic web; these galaxies then settle into violent disk instabilities and end up with dense objects from the dissipative compaction of gas and subsequent starburst (Dekel & Burkert 2014; Zolotov et al. 2015).

We remark that at z > 4, dusty SFGs (DSFGs) display very compact far-IR-emitting regions that show the locations of ongoing starburst and establish a good proxy for the subsequent stellar remnant (e.g., Ikarashi et al. 2015; Oteo et al. 2017; Gómez-Guijarro et al. 2018). In a sample of six DSFGs at z ∼ 4.5 with evidence of minor mergers, Gómez-Guijarro et al. (2018) measured a median stellar mass of $\mathrm{log}({M}_{\star }/{M}_{\odot })=10.49\,\pm 0.32$ and far-IR sizes of ${r}_{e}=0.70\pm 0.29\,\mathrm{kpc}$ . They expect the starburst to be completed in ∼50 Myr, faster than the anticipated timescale for observed mergers of ∼500 Myr. Massive QGs at z ∼ 4 studied here may have stopped star formation earlier (z > 5) than these DSFGs; however, they present the capability of quickly building up and quenching massive stellar cores at such high redshift. Further detailed studies of DSFGs with ALMA are awaited.

We finally quote Hopkins et al. (2010), who reported that the maximum stellar surface densities of GCs and high-z compact QGs are at the global stellar mass surface density limit regardless of their masses and proposed that it is limited by feedback from young massive stars when star formation reaches the Eddington limit. Their results also imply that the densest objects are formed in the extreme situation, which may be only achievable in the early universe.

6.2. Size Evolution on the Evolving Stellar Mass Track

The size–redshift relation of massive QGs in Figure 8 is measured at a fixed stellar mass. Galaxies grow with time, and the evolving cumulative number density determined in the semi-empirical approach using abundance matching has been used to find the progenitors of particular descendants (e.g., Behroozi et al. 2013). Marchesini et al. (2014) tracked the progenitors of ultramassive galaxies today (M_⋆ ≈ 10^11.8 M_⊙) with this method and found their stellar mass evolution as a function of redshift, $\mathrm{log}({M}_{\star }/{M}_{\odot })=A+{Bz}+{{Cz}}^{2},$ where A = 11.801 ± 0.038, B = −0.304 ± 0.054, and C = 0.039 ± 0.014. This relation is based on the observations at z < 3, but if we extrapolate it to z ∼ 4, we find that massive QGs at z ∼ 4 in this study are on this evolutionary track, i.e., they plausibly evolve into ultramassive galaxies today.

Taking the stellar mass evolution into account using the stellar mass–redshift relation in Marchesini et al. (2014), we show the size–stellar mass evolution from massive QGs at z = 4 in Figure 10. The r_e,maj are the median and the 25%–75% interval of the r_e,maj of galaxies with M_⋆ = 10^11.8 M_⊙ from Guo et al. (2009) at z = 0, and extrapolated from the size–stellar mass relation at each redshift in van der Wel et al. (2014) at 0.25 ≤ z ≤ 2.75. The r_e,maj at z > 3 are the observed values since their stellar masses are on the stellar mass–redshift relation of Marchesini et al. (2014). The point at z = 3.1 (Kubo et al. 2017) is not included in the fit; this is shown just for reference. The top and bottom panels of Figure 11 show the size evolution as functions of redshift and cosmic time, respectively. The size–redshift relation is fitted in the form r_e,maj kpc⁻¹ = A × (1+z)^B, where A = 44.1 ± 6.1 and B = −2.6 ± 0.2 or ${r}_{e,{maj}}\,{\mathrm{kpc}}^{-1}=A\times {B}^{-(1+z)}$ , where A = 69.7 ± 7.7 and B = 2.7 ± 0.1. If we fit the size–time relation in the form of $\mathrm{log}({r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1})=A+B\mathrm{log}(t\,{\mathrm{Gyr}}^{-1})$ , we obtain A = −0.56 ± 0.07 and B = 1.91 ± 0.09. We find that they grow in size by a factor of ∼10 in the first few Gyr and finally acquire a size ∼30 times larger than that of a massive QG at z = 4 by z = 0. They have also evolved significantly in the stellar mass surface density (Figure 9).

**Figure 10.** Size–stellar mass growth from massive QGs at z = 4 taking stellar mass evolution into account based on Marchesini et al. (2014). The r_e,maj at each point are those extrapolated from van der Wel et al. (2014, red filled circles), the median, and the 25%–75% interval of the r_e,maj of galaxies with M_⋆ = 10^11.8 M_⊙ at z = 0 from Guo et al. (2009, red unfilled circle), and observed values (others). The black dotted curve shows the best-fit curve. The gray solid and dashed curves show the toy models of size–stellar mass growth in cases of minor mergers ( ${r}_{e,\mathrm{maj}}\propto {M}_{\star }^{2}$ ) and major mergers ( ${r}_{e,\mathrm{maj}}\propto {M}_{\star }$ ), respectively.
Download figure:
Standard image High-resolution image

**Figure 10.** Size–stellar mass growth from massive QGs at z = 4 taking stellar mass evolution into account based on Marchesini et al. (2014). The r_e,maj at each point are those extrapolated from van der Wel et al. (2014, red filled circles), the median, and the 25%–75% interval of the r_e,maj of galaxies with M_⋆ = 10^11.8 M_⊙ at z = 0 from Guo et al. (2009, red unfilled circle), and observed values (others). The black dotted curve shows the best-fit curve. The gray solid and dashed curves show the toy models of size–stellar mass growth in cases of minor mergers ( ${r}_{e,\mathrm{maj}}\propto {M}_{\star }^{2}$ ) and major mergers ( ${r}_{e,\mathrm{maj}}\propto {M}_{\star }$ ), respectively.
Download figure:
Standard image High-resolution image

**Figure 11.** Top: size–redshift relation of massive QGs taking the stellar mass evolution into account. The data points are the same as those in Figure 10. The black solid curve and dashed line show the best-fit curves in forms of ${r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1}\,=A\times {(1+z)}^{B}$ and ${r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1}=A\times {B}^{-(1+z)}$ , respectively. Bottom: similar to the top panel but scaled in cosmic time, and the black solid curve shows the best-fit curve in the form of $\mathrm{log}({r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1})=A+B\mathrm{log}(t\,{\mathrm{Gyr}}^{-1})$ .
Download figure:
Standard image High-resolution image

**Figure 11.** Top: size–redshift relation of massive QGs taking the stellar mass evolution into account. The data points are the same as those in Figure 10. The black solid curve and dashed line show the best-fit curves in forms of ${r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1}\,=A\times {(1+z)}^{B}$ and ${r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1}=A\times {B}^{-(1+z)}$ , respectively. Bottom: similar to the top panel but scaled in cosmic time, and the black solid curve shows the best-fit curve in the form of $\mathrm{log}({r}_{e,\mathrm{maj}}\,{\mathrm{kpc}}^{-1})=A+B\mathrm{log}(t\,{\mathrm{Gyr}}^{-1})$ .
Download figure:
Standard image High-resolution image

In order to constrain the physical processes driving this rapid evolution, we compared the size–stellar mass growth to the two toy models shown in Figure 10. We show size–stellar mass growth models via minor mergers (gray solid curve) and major mergers (gray dashed curve), which follow r ∝ M² and $r\propto M$ , respectively (Bezanson et al. 2009; Naab et al. 2009). The observed size–stellar mass growth closely follows that of minor mergers: it is fit by r_e,maj kpc⁻¹ = A × (M_⋆/10¹¹ M_⊙)^B + C, where A = 1.0 ± 0.4, B = 1.9 ± 0.2, and C = −1.3 ± 0.6 (black dotted curve). Similarly, van Dokkum et al. (2010) evaluated the size–stellar mass evolution of massive galaxies with M_⋆ ≈ 10^11.45 M_⊙ at z = 0, taking the stellar mass evolution into account, using the constant number density method. Although the samples and methods are different, they report an ${r}_{e}\propto {M}^{2.08}$ evolution, similar to our result. The dominant role of dry mergers at high masses in size growth is also reported by the size-age analysis of QGs in Fagioli et al. (2016). Our result also agrees with the prediction of the stellar mass and size growth of QGs on the massive end with M_⋆ ≈ 10^11.8 M_⊙ in Genel et al. (2018) based on the IllustrisTNG simulation. Taking all of these results together, we conclude that the evolution of galaxies on the massive end from z = 4 is likely to be driven by minor mergers.

Note that lower mass galaxies may not necessarily follow the size growth found in this study. Mass-dependent evolution has been predicted in cosmological numerical simulations. For example, more moderate size growth of lower mass galaxies is predicted in Genel et al. (2018). The continuous addition of massive galaxies to the quiescent population, the so-called progenitor bias, may also contribute to the observed size growth (e.g., Carollo et al. 2013; Poggianti et al. 2013) though it alone may not be sufficient (Belli et al. 2015). Several studies reported that the observed merger rate is not capable of the size growth of high-z compact ellipticals (Williams et al. 2011; Newman et al. 2012; Man et al. 2016), but on the other hand, in situ star formation in satellites before mergers can push up the size growth amount via minor mergers (Morishita & Ichikawa 2016). It can also happen that the environment of the most massive galaxies is special. A massive compact elliptical at z = 3.1 cited from Kubo et al. (2017) is in a dense group of massive galaxies capable of size growth of at least 10 times. Further studies of not only compact massive QGs themselves but also their environment are needed to understand what physical processes govern the size–stellar mass growth.

7. Conclusion

We have measured the rest-frame optical sizes of massive galaxies with suppressed star formation at z ∼ 4 with IRCS and AO188 on the Subaru telescope. Although our measurements on individual galaxies are noisy, the more robust size measurements on the stacked object reveals that they have smaller physical sizes compared to lower redshift ones. This is the first measurement of the rest-frame optical sizes of QGs at z ∼ 4. Their mean stellar mass surface density is similar to those of GCs, the densest objects of the universe, although their masses differ by several orders of magnitude. This implies that the origin of the densest galaxies is due to the high density and high gas fraction in the early universe. If we take the stellar mass evolution into account, they plausibly evolve into the most massive galaxies today, and their stellar mass–size evolution is consistent with a scenario in which minor dry mergers drive size growth.

We have shown that massive QGs at z ∼ 4 are compact, but we have pushed the ability of current facilities close to the limit. Deeper and higher resolution imaging at >2 μm with AO on large(r) ground-based telescopes and the James Webb Space Telescope (JWST) is needed to make a leap from here.

M.T. acknowledges support by JSPS KAKENHI grant No. 15K17617. K.Y. was supported by JSPS KAKENHI grant Nos. JP16K17659 and JP18K13578. M.S. and S.T. acknowledge support from the European Research Council (ERC) Consolidator Grant funding scheme (project ConTExt, grant number 648179). The Cosmic Dawn Center is funded by the Danish National Research Foundation. This work is based on data collected at the Subaru Telescope, which is operated by the National Astronomical Observatory of Japan. We thank the anonymous referee for the useful report, which helped improve the paper.

The Rest-frame Optical Sizes of Massive Galaxies with Suppressed Star Formation at z ∼ 4

Article metrics

Permissions

Author e-mails

Author affiliations

ORCID iDs

Dates

Abstract

1. Introduction