We investigate the physical conditions for the growth of intermediate-mass seed black holes assumed to have formed from remnants of the first generation of massive stars. We follow the collapse of high σ halos with T_vir > 10⁴ K using cosmological, smooth-particle hydrodynamic (SPH) simulations in the standard ΛCDM model. During the collapse of the parent halo, the seed holes are incorporated through mergers into larger systems and accrete mass from the surrounding gas. We include a self-consistent treatment of star formation, black hole accretion, and associated feedback processes. Even under optimistic assumptions for the seed black hole mass and for efficient merger rates, we find that seed holes in halos M ≤ 10¹⁰ M never reach the conditions for critical Eddington growth. Most of the black hole growth in this regime is determined by the initial mass and the merger rates. Critical accretion rates are reached, albeit only after a significant delay, at the time of collapse (z ~ 7) for 3-4 σ halos of M ~ 10¹¹ M. Our results imply M_BH = 5 × 10⁶ M (M_halo/10¹¹ M)^0.78 at the time of collapse. The required conditions of Eddington growth to explain the buildup of supermassive black holes (~10⁹ M), as implied by Sloan quasars at z > 6, are therefore hard to meet in such a scenario. Without a "jump start," these conditions may be only achieved in extremely rare halos with M_halo > 10¹³ that collapsed before z ~ 6. The sub-Eddington regime in which black holes accrete at early time implies a small contribution to the reionization by miniquasar but still sufficient to cause appreciable heating of the IGM at z 15-18.