Precise lung tumor localization in real time is particularly important for some motion management techniques, such as respiratory gating or beam tracking with a dynamic multi-leaf collimator, due to the reduced clinical tumor volume (CTV) to planning target volume (PTV) margin and/or the escalated dose. There might be large uncertainties in deriving tumor position from external respiratory surrogates. While tracking implanted fiducial markers has sufficient accuracy, this procedure may not be widely accepted due to the risk of pneumothorax. Previously, we have developed a technique to generate gating signals from fluoroscopic images without implanted fiducial markers using a template matching method (Berbeco et al 2005 Phys. Med. Biol. 50 4481–90, Cui et al 2007 Phys. Med. Biol. 52 741–55). In this paper, we present an extension of this method to multiple-template matching for directly tracking the lung tumor mass in fluoroscopy video. The basic idea is as follows: (i) during the patient setup session, a pair of orthogonal fluoroscopic image sequences are taken and processed off-line to generate a set of reference templates that correspond to different breathing phases and tumor positions; (ii) during treatment delivery, fluoroscopic images are continuously acquired and processed; (iii) the similarity between each reference template and the processed incoming image is calculated; (iv) the tumor position in the incoming image is then estimated by combining the tumor centroid coordinates in reference templates with proper weights based on the measured similarities. With different handling of image processing and similarity calculation, two such multiple-template tracking techniques have been developed: one based on motion-enhanced templates and Pearson's correlation score while the other based on eigen templates and mean-squared error. The developed techniques have been tested on six sequences of fluoroscopic images from six lung cancer patients against the reference tumor positions manually determined by a radiation oncologist. The tumor centroid coordinates automatically detected using both methods agree well with the manually marked reference locations. The eigenspace tracking method performs slightly better than the motion-enhanced method, with average localization errors less than 2 pixels (1 mm) and the error at a 95% confidence level of about 2–4 pixels (1–2 mm). This work demonstrates the feasibility of direct tracking of a lung tumor mass in fluoroscopic images without implanted fiducial markers using multiple reference templates.