Recent studies have shown that massive quiescent galaxies at high redshift are much more compact than present-day galaxies of the same mass. Here we compare the radial stellar density profiles and the number density of a sample of massive galaxies at z ~ 2.3 to nearby massive elliptical galaxies. We confirm that the average stellar densities of the z ~ 2.3 galaxies within the effective radius, ρ(<r_e ), are two orders of magnitude higher than those of local elliptical galaxies of the same stellar mass. However, we also find that the densities measured within a constant physical radius of 1 kpc, ρ(<1 kpc), are higher by a factor of 2-3 only. This suggests that inside-out growth scenarios are plausible, in which the compact high-redshift galaxies make up the centers of normal nearby ellipticals. The compact galaxies are common at high redshift, which enables us to further constrain their evolution by requiring that the number density of their descendants does not exceed constraints imposed by the z = 0 galaxy mass function. We infer that size growth must be efficient, with (r ₁₊₂/r ₁) ~ (M ₁₊₂/M ₁)². A simple model where compact galaxies with masses ~10¹¹ M _☉ primarily grow through minor mergers produces descendants with the approximate sizes, stellar densities, and number density of elliptical galaxies with masses 2-3 × 10¹¹ M _☉ in the local universe. We note that this model also predicts evolution in the M _BH – σ relation, such that the progenitors of elliptical galaxies have lower black hole masses at fixed velocity dispersion. The main observational uncertainty is the conversion from light to mass; measurements of kinematics are needed to calibrate the masses and stellar densities of the high-redshift galaxies.