We consider the nature of the elusive neutron star candidate 1E 161348-5055 using X-ray observations obtained with ASCA. The compact X-ray source is centered on the shell-type Galactic supernova remnant RCW 103 and has been interpreted as a cooling neutron star associated with the remnant. The X-ray spectrum of the remnant shell can be characterized by a nonequilibrium ionization (NEI) thermal model for a shocked plasma of temperature kT ~0.3 keV. The spectrum falls off rapidly above 3 keV to reveal a point source in the spectrally resolved images, at the location of 1E 161348-5055. A blackbody model fit to the source spectrum yields a temperature kT=0.6 keV, with an unabsorbed 0.5-10 keV luminosity of L_X ~10³⁴ ergs s⁻¹ (for an assumed distance of 3.3 kpc), both of which are at least a factor of 2 higher than predicted by cooling neutron star models. Alternatively, a power-law model for the source continuum gives a steep photon index of α ~3.2, similar to that of other radio-quiet, hard X-ray point sources associated with supernova remnants. 1E 161348-5055 may be prototypical of a growing class of radio-quiet neutron stars revealed by ASCA; we suggest that these objects account for previously hidden neutron stars associated with supernova remnants.