There is growing evidence that the estimations of the beaming Doppler factor in TeV BL Lac objects based on the synchrotron self-Compton (SSC) models are in strong disagreement with those deduced from the unification models between blazars and radio galaxies. When corrected from extragalactic absorption by the diffuse infrared background (DIrB), the SSC one-zone models require a very high Lorentz factor (around 50) to avoid strong γ-γ absorption. However, the statistics on beamed versus unbeamed objects, as well as the luminosity contrast, favors a much lower Lorentz factor, on the order of 3. In this paper, we show that for the special case of Markarian 501, the need for a very high Lorentz factor is unavoidable for all one-zone models in which all photons are assumed to be produced at the same location at the same time. Models assuming a double structure with two different beaming patterns can partially solve the problem of luminosity contrast, but we point out that they are inconsistent with the statistics on the number of detected TeV sources. The only way to solve the issue is to consider inhomogeneous models, in which low-energy and high-energy photons are not produced at the same place, allowing for much smaller Lorentz factors. This approach implies that the jet is stratified, but also that the particle energy distribution is close to a monoenergetic one and that pair production is likely to be significant. The implications for relativistic jet physics and the particle acceleration mechanism are discussed.