The Event Horizon Telescope, a global 230 GHz very-long-baseline interferometry array, achieves angular resolution of , sufficient to resolve the supermassive black hole Sagittarius A* (Sgr A*). This resolution may soon enable measurements of the black hole "shadow" size and asymmetry, predicted to be ≈50 and ≲3 μas, respectively. Measurements that depart from these values could indicate a violation of the "no-hair theorem." However, refractive scattering by the turbulent ionized interstellar medium distorts the image of Sgr A*, affecting its apparent size and asymmetry. In this paper, we present a general analytic approach to quantify the expected image wander, distortion, and asymmetry from refractive scattering. If the turbulence in the scattering material of Sgr A* is close to Kolmogorov, we estimate the mean refractive image wander, distortion, and asymmetry to be 0.53, 0.72, and 0.52 μas at 230 GHz. However, alternative scattering models with flatter power spectra can yield larger values, up to 2.1, 6.3, and 5.0 μas, respectively. We demonstrate that these effects can be reduced by averaging images over multiple observations. For a small number of observations, the effects of scattering can be comparable to or greater than those from black hole spin, and they determine a fundamental limit for testing general relativity via images of Sgr A*.