We use very deep U_nGI multifield imaging obtained at the Keck telescope to study the evolution of the rest-frame 1700 Å galaxy luminosity function as the universe doubles its age from z ~ 4 to ~2. We use exactly the same filters and color-color selection as those used by the Steidel team but probe significantly fainter limits, well below L*. The depth of our imaging allows us to constrain the faint end of the luminosity function, reaching M₁₇₀₀ ~ -18.5 at z ~ 3 (equivalent to ~1 M yr^-1), accounting for both N^1/2 uncertainty in the number of galaxies and cosmic variance. We carefully examine many potential sources of systematic bias in our LF measurements before drawing the following conclusions. We find that the luminosity function of Lyman break galaxies evolves with time and that this evolution is differential with luminosity. The result is best constrained between the epochs at z ~ 4 and ~3, where we find that the number density of sub-L* galaxies increases with time by at least a factor of 2.3 (11 σ statistical confidence); while the faint end of the LF evolves, the bright end appears to remain virtually unchanged, indicating that there may be differential, luminosity-dependent evolution (98.5% statistical probability). Potential systematic biases restrict our ability to draw strong conclusions about continued evolution of the luminosity function to lower redshifts, z ~ 2.2 and ~1.7, but, nevertheless, it appears certain that the number density of z ~ 2.2 galaxies at all luminosities we studied, -22 > M₁₇₀₀ > -18, is at least as high as that of their counterparts at z ~ 3. While it is not yet clear what mechanism underlies the observed evolution, the fact that this evolution is differential with luminosity opens up new avenues of improving our understanding of how galaxies form and evolve at high redshift.