We extend the standard theory of cosmological perturbations to homogeneous but anisotropic universes. We present an exhaustive computation for the case of a Bianchi I model, with a residual isotropy between two spatial dimensions, which is undergoing complete isotropization at the onset of inflation; we also show how the computation can be further extended to more general backgrounds. In the presence of a single inflaton field, there are three physical perturbations (precisely as in the isotropic case), which are obtained (i) by removing gauge and non-dynamical degrees of freedom, and (ii) by finding the combinations of the remaining modes in terms of which the quadratic action of the perturbations is canonical. The three perturbations, which later in the isotropic regime become a scalar mode and two tensor polarizations (gravitational wave), are coupled to each other already at the linearized level during the anisotropic phase. This generates non-vanishing correlations between different modes of the cosmic microwave background (CMB) anisotropies, , which can be particularly relevant at large scales (and, potentially, be related to the large scale anomalies in the WMAP (Wilkinson Microwave Anisotropy Probe) data). As an example, we compute the spectrum of the perturbations in this Bianchi I geometry, assuming that the inflaton is in a slow roll regime also in the anisotropic phase. For this simple set-up, fixing the initial conditions for the perturbations appears more difficult than in the standard case, and additional assumptions seem to be needed to provide predictions for the CMB anisotropies.