Soft gamma-ray repeaters (SGRs) and anomalous X-ray pulsars (AXPs) are young and radio-quiet X-ray pulsars that have been rapidly spun-down to slow spin periods clustered in the range 5-12 s. Most of these unusual pulsars also appear to be associated with supernova shell remnants (SNRs) with typical ages less than 30 kyr. By examining the sizes of these remnants versus their ages, we demonstrate that the interstellar media that surrounded the SGR and AXP progenitors and their SNRs were unusually dense compared to the environments around most young radio pulsars and SNRs. We explore the implications of this evidence on magnetar and propeller-based models for the rapid spin-down of SGRs and AXPs. We find that evidence of dense environments is not consistent with the magnetar model unless a causal link can be shown between the development of magnetars and the external interstellar medium. Propeller-driven spin-down by fossil accretion disks for SGRs and AXPs appears to be consistent with dense environments since the environment can facilitate the formation of such a disk. This may occur in two ways: (1) formation of a "pushback" disk from the innermost ejecta pushed back by prompt reverse shocks from supernova remnant interactions with massive progenitor wind material stalled in dense surrounding gas or (2) acquisition of disks by a high-velocity neutron stars, which may be able to capture sufficient amounts of comoving outflowing ejecta slowed by the prompt reverse shocks in dense environments.