We show that the X-ray and γ-ray spectra of Swift GRBs and their afterglows are consistent with the emission characteristic of an expanding, relativistic fireball. The classical afterglow due to the impact of the fireball on the external medium is often not observed until 1 to several hours after the GRB. Focusing on GRBs 061121, 060614, and 060124, but generalizing to the full (>50 Ms XRT exposure) Swift sample up to and including GRB 061210, we show that the early emission in >90% of early afterglows has a characteristic νF_ν spectral energy E_peak, which likely evolves from the γ-rays through the soft X-ray band on timescales of 10²-10⁴ s after the GRB. The observed spectra are strongly curved when plotted with logarithmic axes and have often been incorrectly fitted in other studies with a time-varying soft X-ray absorption. The spectral evolution inferred from fitting instead models used to fit GRBs demonstrates a common evolution—a power-law hardness-intensity correlation and hard-to-soft evolution—for GRBs and the early X-ray afterglows and X-ray flares. Combined with studies of short-timescale variability, our findings indicate a central engine active for longer than previously suspected. The GRB spectra are observed to become very soft at late times due to an intrinsic spectral evolution and due to the surprising faintness of some afterglows. We discuss models for the early X-ray emission.