A second-order strain gradient nonlocal shell model is established to study the mode transformation in single-walled carbon nanotubes (SWCNTs). Nonlocal length is calibrated carefully for SWCNTs in reference to molecular dynamics (MD) simulations through analysis of nonlocal length effects on the frequencies of the radial breathing mode (RBM) and circumferential flexural modes (CFMs) and its effects on mode transformation. All analyses show that only a negative second-order nonlocal shell model is appropriate to SWCNTs. Nonlocal length is evidently related to vibration modes and the radius-to-thickness ratio. It is found that a nonlocal length is approximately 0.1 nm in an average sense when RBM frequency is concerned. A nonlocal length of 0.122–0.259 nm is indicated for the mode transformation in a selected group of armchair SWCNTs. 2:1 and 1:1 internal resonances are found for the same SWCNT based on different models, which implies that the internal resonance mechanism depends on the model employed. Furthermore, it is shown that an effective thickness of approximately 0.1 nm is more appropriate to SWCNTs than 0.066 nm.