In this paper, we characterize 3D optical imaging techniques as 3D linear shift-invariant filtering operations. From the Helmholtz equation that is the basis of scalar diffraction theory, we show that the scattered field, or indeed a holographic reconstruction of this field, can be considered to be the result of a linear filtering operation applied to a source distribution. We note that if the scattering is weak, the source distribution is independent of the scattered field and a holographic reconstruction (or in fact any far-field optical imaging system) behaves as a 3D linear shift-invariant filter applied to the refractive index contrast (which effectively defines the object). We go on to consider tomographic techniques that synthesize images from recordings of the scattered field using different illumination conditions. In our analysis, we compare the 3D response of monochromatic optical tomography with the 3D imagery offered by confocal microscopy and scanning white light interferometry (using quasi-monochromatic illumination) and explain the circumstances under which these approaches are equivalent. Finally, we consider the 3D response of polychromatic optical tomography and in particular the response of spectral optical coherence tomography and scanning white light interferometry.