A novel design of an Earth field digital fluxgate magnetometer is presented, the small magnetometer in low-mass experiment (SMILE). The combination of a number of new techniques results in significant miniaturization of both sensor and electronics. The design uses a sensor with volume compensation, combining three dual rod cores in a Macor® cube with the side dimension of 20 mm. Use of volume compensation provides high geometrical stability of the axes and improved performance compared to component compensated sensors. The sensor is operated at an excitation frequency of 8 kHz. Most of the instrument functionality is combined in a digital signal processing core, implemented in a field programmable gate array (FPGA). The pick-up signal is digitized after amplification and filtering, and values of compensation currents for each of the axes are determined by a digital correlation algorithm, equivalent to a matched filter, and are fed to a hybrid pulse-width modulation/delta-sigma digital-to-analogue converter driving the currents through the compensation coils. Using digital design makes the instrument very flexible, reduces power consumption and opens possibilities for the customization of the operation modes. The current implementation of the design is based on commercial off-the-shelf components. A calibration of the SMILE instrument was carried out at the Nurmijärvi Geophysical Observatory, showing high linearity (within 6 nT on the whole ±50 µT scale), good orthogonality (22 arcmin) and very good temperature stability of the axes.