The characteristic size of early-type galaxies (ETGs) of given stellar mass is observed to increase significantly with cosmic time, from redshift z 2 to the present. A popular explanation for this size evolution is that ETGs grow through dissipationless ("dry") mergers, thus becoming less compact. Combining N-body simulations with up-to-date scaling relations of local ETGs, we show that such an explanation is problematic, because dry mergers do not decrease the galaxy stellar-mass surface density enough to explain the observed size evolution, and also introduce substantial scatter in the scaling relations. Based on our set of simulations, we estimate that major and minor dry mergers increase half-light radius and projected velocity dispersion with stellar mass as R _e M ^{1.09 ± 0.29} _* and σ_e2 M ^{0.07 ± 0.11} _*, respectively. This implies that: (1) if the high-z ETGs are indeed as dense as estimated, they cannot evolve into present-day ETGs via dry mergers; (2) present-day ETGs cannot have assembled more than ~45% of their stellar mass via dry mergers. Alternatively, dry mergers could be reconciled with the observations if there was extreme fine tuning between merger history and galaxy properties, at variance with our assumptions. Full cosmological simulations will be needed to evaluate whether this fine-tuned solution is acceptable.