The characterization of deep surface trenches with atomic force microscopy (AFM) presents significant challenges due to the sharp step edges that disturb the instrument and prevent it from faithfully reproducing the sample topography. Previous authors have developed AFM methodologies to successfully characterize semiconductor surface trenches with dimensions on the order of tens of nanometers. However, the study of imaging fidelity for features with dimensions smaller than 10 nm has not yet received sufficient attention. Such a study is necessary because small features in some cases lead to apparently high-quality images that are distorted due to tip and sample mechanical deformation. This paper presents multi-scale simulations, illustrating common artifacts affecting images of nanoscale trenches taken with fine carbon nanotube probes within amplitude-modulation and frequency-force-modulation AFM (AM-AFM and FFM-AFM, respectively). It also describes a methodology combining FFM-AFM with a step-in/step-out algorithm analogous to that developed by other groups for larger trenches, which can eliminate the observed artifacts. Finally, an overview of the AFM simulation methods is provided. These methods, based on atomistic and continuum simulation, have been previously used to study a variety of samples including silicon surfaces, carbon nanotubes and biomolecules.