Swift heavy ions cause material modification along their tracks, changes primarily due to their very dense electronic excitation. The available data for threshold stopping powers indicate two main classes of materials. Group I, with threshold stopping powers above about 10 keV nm⁻¹, includes some metals, crystalline semiconductors and a few insulators. Group II, with lower thresholds, comprises many insulators, amorphous materials and high T_c oxide superconductors. We show that the systematic differences in behaviour result from different coupling of the dense excited electrons, holes and excitons to atomic (ionic) motions, and the consequent lattice relaxation. The coupling strength of excitons and charge carriers with the lattice is crucial. For group II, the mechanism appears to be the self-trapped exciton model of Itoh and Stoneham (1998 Nucl. Instrum. Methods Phys. Res. B 146 362): the local structural changes occur roughly when the exciton concentration exceeds the number of lattice sites. In materials of group I, excitons are not self-trapped and structural change requires excitation of a substantial fraction of bonding electrons, which induces spontaneous lattice expansion within a few hundred femtoseconds, as recently observed by laser-induced time-resolved x-ray diffraction of semiconductors. Our analysis addresses a number of experimental results, such as track morphology, the efficiency of track registration and the ratios of the threshold stopping power of various materials.