Modulation-enhanced localization microscopy

Cell biology and microscopy are closely intertwined since the invention of light microscopy in the late 1600s, attributed to Antonie van Leeuwenhoek who was the first to observe bacteria and protozoa. Fluorescence microscopy—pioneered in the early 1900s—is based on substances which absorb light that is reemitted as fluorescence at a longer wavelength determined by the Stokes shift [1, 2]. Nowadays, fluorescence microscopists in state-of-the-art biolabs can specifically tag and image molecular components of cells and tissues (such as proteins, lipids, DNA, RNA) in various biological systems with minimal invasiveness [2–4]. One of the key developments has been the advent of genetically encoded fluorescent proteins, which enabled to study protein localization, dynamics and function in cells [5, 6]. Over the last two decades, this approach generated fundamental knowledge about cellular processes and profoundly advanced life science research.

Despite the advantages of optical microscopy, both brightfield microscopes (using visible light passing through the sample) and fluorescence-based microscopes are fundamentally limited in spatial resolution by the wave nature of light. Specifically, the optical resolution is set by the diffraction limit of light to around $\lambda /2$, where $\lambda$ is the wavelength of the collected light [7]. For example, considering the emission of a green fluorescent protein (GFP, $\lambda \sim500$ nm), no structure below 250 nm would be resolvable.

Pioneering steps in the direction of super-resolution fluorescence microscopy were set in the late 1980's, with the recording of tracks of single light-scattering objects and fluorescent particles [8–10]. First described by Heisenberg around 1930, this approach used the fact that sparse point emitters can be detected with high precision: each recorded photon carries information about the true location of its—well separated—emitter [11, 12]. The precision of localization is then given by the spreading, called point spread function (PSF), of the photons around the emitter position in the image plane. The PSF of the imaging system is characterized by a standard deviation $s$ set by the numerical aperture (NA) and $\lambda$, the fluorescence emission wavelength, by $s \cong \lambda /{\text{NA}}$. Collecting $N$ photons is equivalent to repeating $N$ independent measurement of the emitter's position, leading to an uncertainty given by the standard error of the mean [13] $dx = s/\sqrt N$. A precision of about 1 nm in the localization of a particle would therefore be achievable with around 10⁴ photons detected, assuming a very low background noise.

In 1995, Eric Betzig published a first rough idea for how super-resolution optical microscopy based on the localization of single fluorescent molecules could be realized [12]. It did, however, take yet another 10 years, before he and colleagues were able to truly demonstrate this concept in 2006 by consecutively photo-activating and localizing single photo-activatable fluorescent proteins and by superimposing all detected single molecule locations to create one pointillistic image [14]. This was the starting point for single molecule localization microcopy (SMLM) methods such as (F)PALM, (d)STORM and µPAINT [15–18].

In a different attempt to overcome the diffraction limit of light, in the late 1990's Stefan Hell proposed stimulated emission depletion (STED) of fluorophores [19, 20]. In 2000, Klar et al showed a 5 to 6-fold increase in the axial resolution of a modified confocal microscope using depletion spots above and below the focused excitation spot. This depletion pattern was generated by placing a phased plate in the beam path of the depletion laser (called STED beam), resulting in a non-linear inhibition of active fluorescence to narrow the effective PSF of the microscope. Even though both the excitation and depletion laser spots are diffraction-limited, their combination results in a non-linear depletion of the fluorophore's emission, which reduces the effective PSF of the excitation volume. After its initial realization, the concept was immediately extended to multiple colors, 2D/3D versions, cell applications and was generalized to reversibly photo-switchable fluorescent labels (RESOLFT) [21–24].

Around the year 2000, another super-resolution microscopy (SRM) concept was introduced by Gustafsson and Heintzmann based on the use of interference patterns in structured illumination microscopy (SIM) [25, 26]. In contrast to PALM, where typically more than 10⁴ raw-images are used to create a super-resolved image, SIM relies on 9 or 15 raw-images to create one super-resolved image. The physical principle of SIM is based on illuminating a sample with a striped pattern that will shift high-frequency image information below the diffraction limit into the lower frequency domain that can be captured by the objective lens [27, 28]. This is highlighted by the so-called Moiré patterns, which carry information about previously inaccessible details of the sample. The advantage of SIM is that super-resolved images are created without the need of special fluorophores since super-resolution linear SIM is not dependent on specific photo-physical dye characteristics, required in methods such as PALM, STORM or STED. The disadvantage of linear SIM is that it is limited to a resolution gain by a factor of two, down to approximately 100 nm. SIM has also been extended to non-linear SIM using photo-switchable dyes, which allows improving the image resolution, achieving about 50 nm lateral resolution [29–31].

In 2017, Balzarotti et al introduced MINFLUX—minimal photon fluxes—in which a radically new concept of SRM was presented with the potential to virtually reach unlimited resolution with minimal photo-damage to the fluorophore [32]. MINFLUX reached 2 nm localization precision by using less than $N = 500$ photons. Although the technical setup uses a doughnut-shaped laser beam, the method is drastically different from STED microscopy. In MINFLUX, single fluorescent emitters are kept close to the center of a doughnut-shaped illumination profile. The idea is to use the well-known doughnut shaped illumination profile as a ruler for probing the emitter's position. The position of the emitter is retrieved by a triangulation localization approach as outlined further below. MINFLUX can achieve better localization than the theoretical limit of SMLM methods and was extended to isotropic 3D localization precision [33, 34].

In 2019, the parallelization of the MINFLUX method was first described by Reymond et al using the acronym SIMPLE [35] and followed by similar approaches [36–38] named ROSE, SIMFLUX and ModLoc. In a recent paper [33], Hell et al also extended the field of view (FOV) of MINFLUX from initially around 100 × 100 nm² to a size of about 10 × 10 µm². In this perspective, we will describe the novel concept of these modulation-enhanced localization microscopies (meLM), its relation to STED, SMLM and SIM, and outline current technical realizations, limitations, and future perspectives of the approach.

In order to demonstrate how meLM is positioned within the field of super-resolution microscopy, we will split the current super-resolution (SR) methods into three main principles: stimulated emission depletion microscopy (STED), single molecule localization microscopy (SMLM) and structured illumination microscopy (SIM) [39–41]. To get an overview over the current SR methods and the novelty and relation of meLM to these three methods (STED, SMLM and SIM), we classify and structure all four approaches as shown in figure 1. Despite some shared features between meLM and the existing SR techniques (such as structured illumination profile or on/off switching of the fluorophores to ensure sample's sparsity), meLM relies on a fundamentally different principle as described below, therefore justifying it to be named a fourth SR microscopy technique. In figure 1, all four SR methods are further split into four quadrants highlighting their specific requirements: (1) Illumination (blue), (2) photo-physics of the fluorophore (green), (3) setup and detection method (red) and (4) image generation, necessary to create the super-resolved image (yellow). Control of illumination and photo-physical properties in each method enable spatial excitation and fluorescence emission control prior to signal detection by the objective lens (blue and green). Detection type and signal analysis in each method describes the handling of emitted photons after passing the objective lens (red and yellow). Note that machine-learning approaches can be combined with existing techniques such as SMLM [42, 43] and SIM [44, 45] for better and faster reconstructions, and we believe meLM will benefit from it too.

Zoom In Zoom Out Reset image size

2.1. STED

2.2. SMLM

2.3. SIM

2.4. meLM

The fundamental requirements of modulation-enhanced localization microscopy are the following:

• a.  
  A well-characterized structured illumination pattern (e.g. sinusoidal wave or doughnut-shaped beam)
• b.  
  The possibility to shift this pattern with high accuracy (sub-nanometric precision achievable with spatial light modulators (SLMs) or electro-optical deflectors (EODs))
• c.  
  A sparse sampling of fluorescently labelled molecules
• d.  
  Single photon counting with high precision and sensitivity

Even though not strictly necessary, one assumes, for simplicity, a linear response of the fluorophore to the excitation; this assumption is reasonable for most of the fluorophores and for laser powers below the saturation intensity for most standard fluorophores. In the case of a sinusoidal illumination pattern, a series of three measurements are performed along the direction of the pattern, each of them corresponding to a different translational position of the pattern [35]. The collected fluorescence intensity, response of the emitter to the excitation, directly depending on its relative position with respect to the illumination pattern, will therefore be modulated as the illumination is shifted. From this series of three images, one can extract three different numbers of emitted photons ${N_i}$, which follow the shape of the illumination profile (in the case of a non-linear response, one simply has to combine the response with the excitation pattern). It is important to assume a stable photo-physical behavior during the measurement time. From these modulated emission intensities ${N_i}$, one retrieves the position of the emitter through fitting of the illumination's shape; a sinusoidal function in this case. As x-y positions can be determined independently, using two orthogonal illumination patterns is sufficient for 2D localization of a single particle. The procedure is therefore repeated after a 90º rotation of the illumination pattern. MINFLUX works in a similar way, using a doughnut-shaped illumination and consecutive measurements at four positions to retrieve a x-y localization. This approach is radically different from SIM and SMLM as it relies neither on image reconstruction of Moiré patterns in Fourier space, nor on centroid fitting of diffraction-limited photon distributions.

The localization precision of meLM can be evaluated using error propagation and the Cramer–Rao Lower Bound (CRLB) as derived for sinusoidal, Gaussian and quadratic functions [32]. In the next section, we will describe the resulting localization precision by using either large or small pattern shifts, where we achieve a homogeneous $\sim$2-fold localization improvement or higher localization gains, respectively.

Altogether, the concept relies on phase-shifting of the otherwise stable illumination pattern which can be performed with nanometer precision ($<$1 nm) to probe the actual particle position through photon number variations for different phase shifts. The measured photon counts can be fitted to the known illumination pattern to derive the actual particle location leading to higher localization precisions compared to centroid fitting in classical SMLM.

The power of the newly developed meLM technique comes from the extra information one can extract from the structured nature of the previously well-characterized illumination pattern. In other words, by putting into the system more knowledge—in this case the illumination’s spatial shape and spatial shifts—one can achieve higher localization precision. The requirement is that there is a technology to control illumination patterns and shifts at precisions significantly smaller than the wavelength of the emitted light. This, however, can be achieved easily e.g. by displacing the sample underneath a stationary pattern using high-precision piezo-position stages, or by displacing the pattern. MINFLUX, which opened the path for this class of techniques, directly exemplifies an extreme case, where the emitters are localized in the minimum of the doughnut-shaped illumination profile using sub-nanometer shifts. However, this is only possible at the expense of the size of the field of view, as only the particles resting at the center of the doughnut will be localized with higher precision (see below). In comparison, other techniques such as SIMPLE, ROSE, SIMFLUX and iterative MINFLUX have the advantage of providing a localization precision improvement over large fields of view and have been shown to yield an improvement factor of around two.

The key difference between meLM and other SR techniques is that information about the actual emitter’s position can be extracted even in the complete absence of any detected photons in an image. Indeed, by considering a region of the FOV, where nothing but noise is visible on a camera frame, only two plausible conclusions can be drawn:

• (1)  
  There is no emitter at this location.
• (2)  
  The emitter is located in a trough with a zero value of the illumination intensity.

The latter hypothesis is then answered to with the next acquired image where the illumination is shifted: a brightness increase will be observed if there is an emitter present. This Gedankenexperiment shows that after a full measurement cycle, i.e. successive phase shifts of the illumination, both the ‘absence of’ and the actual collected photons contributed to defining the position of the emitter. This contrasts with standard SMLM that would only employ the directly captured photon information.

In the following subsections, we will describe how this homogeneous two-fold increase in localization precision is achieved, as well as the more specific regimes where the gain becomes virtually infinite using small pattern shifts.

3.1. Homogeneous two-fold localization precision gain

Whenever (1) the average illumination intensity is equal, and (2) the ‘modulation efficiency’ (defined further below) is constant over the FOV, a homogeneous resolution gain over the whole FOV will be reached:

• (1)  
  Assuming a sinusoidal illumination described by:

$$\begin{equation*}I\left( x \right) = \frac{A}{2}\left( {1 + m{\text{ cos}}\left( {\frac{{2\pi }}{\lambda }\left( {x - {\phi _0}} \right)} \right)} \right)\end{equation*}$$

with $A$ the amplitude, ${\phi _0}$ the phase and $m$ the modulation depth. Taking $k \ge 2$ equally spaced sampling points on the illumination, i.e. ${x_i} = \frac{1}{k}\left( {i - \frac{{k + 1}}{2}} \right), i = 1, \ldots ,k$, one obtains an expression for the average total number of photons:$R$

$$\begin{equation*}N = \mathop \sum \limits_{i = 1}^k {n_i} = \mathop \sum \limits_{i = 1}^k R\left( {I\left( {{x_i}} \right)} \right).\end{equation*}$$

R represents the response function of the fluorophore to a given excitation, generalizing the approach to any illumination pattern and linear/non-linear response of the fluorophore. In the following, this response is assumed to be linear, without loss of generality $R = \mathbb{1}$, leading to:

$$\begin{equation*}N = \mathop \sum \limits_{i = 1}^k \frac{A}{2}\left( {1 + m{\text{ cos}}\left( {\frac{{2\pi }}{\lambda }\left( {{x_i} - {\phi _0}} \right)} \right)} \right) = \frac{{2A}}{k}.\end{equation*}$$

Please note that the resulting expression does not depend on ${\phi _0}$, and is therefore constant over the FOV whenever$k \ge 2$.

• (2)  
  Next, we introduce a 'modulation efficiency'${\varepsilon _m}$, defined as the normalized standard deviation of ${n_i}$ the recorded number of photons:

  $$\begin{equation*}{\varepsilon _m} = \frac{{{\text{std}}\left( {\left\{ {{n_i}} \right\}} \right)}}{N} = \left\{ { \begin{array}{*{20}{c}} {\frac{m}{{\sqrt 2 }} \left| {{\text{sin}}\left( {\frac{{2\pi }}{\lambda }{\phi _0}} \right)} \right|, \quad\quad\text{if}\,\, k = 2} \\ {\frac{m}{{\sqrt {2 k\left( {k - 1} \right)} }}, \quad\quad\quad\quad\quad\text{if}\,\, k > 2.} \end{array}} \right.\end{equation*}$$

If the number of sampling points is below three, the modulation efficiency will exhibit a spatial dependence. For three or more frames per pattern orientation, a homogeneous modulation can be achieved, necessary for a homogeneous precision gain. Even though demonstrated for the specific case of a sinusoidal illumination pattern, this reasoning will hold for any arbitrary illumination pattern.

A two-fold localization precision gain over the field of view has been verified both in silico and experimentally by different groups [35–38]. The localization gain is defined by comparing the localization precision of meLM to SMLM for a given number of photons. Figure 2 shows one measurement to determine the position of a single emitter using the SIMPLE method. First, three intensities are recorded at three illumination pattern positions and second, a fit to the illumination pattern retrieves the actual single molecule position. Figure 3 shows the repetition of this process to get information about the statistical behavior of the three fit parameters: the amplitude, phase and offset. The photon spreading, shot-noise and read-out noise are included in these simulations showing a Poissonian-like distributed variability of photon counts [53] matching $\sqrt N$. We assume that the phase shift of the pattern is infinitely accurate, there is no movement or drift of the emitter and the laser has no intensity variation during the time scale of the measurement. The crucial parameter is the phase φ, as it corresponds to the position of the emitter on the sinusoidal wave pattern, i.e. its location. The spreading of the recovered phase ${\sigma _\phi }$ directly gives the localization precision. Figure 3(b) shows the comparison of the theoretical limit of SMLM as derived by Mortensen et al 2010 (revised formula from Thompson et al 2002) [13, 54]. The SIM based localization techniques, such as SIMPLE, provide a two-fold improvement in localization precision, homogeneously over the FOV and regardless of the number of recorded photons.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

3.2. Extreme localization precision gain: minimal photon flux

In the previous section, we described a shift of the excitation pattern by a third of the illumination pattern's wavelength, resulting in a high modulation of the emitter's fluorescence response following the full excitation amplitude. As previously mentioned, by reducing the displacement of the illumination pattern, one can reach even greater gain in localization precision: the fluorescent emitters present in the valley of the excitation will be precisely localized even in near-absence of photon emission. In such a case, the precision arises from the fact that an emitter can be kept in the zero-intensity region of the illumination pattern by shifting and registering its displacements with sub-nanometric precision.

Where SMLM uses the fluorescence signal to statistically retrieve the position of the centroid, for each photon collected, meLM extracts the information about the likelihood of an emitter to be present at a certain distance from the excitation's valley. In short, the paradigm changes from 'Where is the emitter in the FOV?' to 'Is it located in the valley?'. By narrowing down the scope of the question, each individual answer (photon counted) gives rise to a more accurate information. This enhanced precision comes at the expanse of the covered surface, as this gain exists only within the dark regions of the illumination.

By adding a certain background noise level on top of the emitted fluorescence, the principle can still be applied, and a high localization precision can be envisioned. In addition, reduced modulation contrast due to limitations in real setup configurations can be accounted for with an increased background noise as described in Reymond et al [35].

Therefore, for smaller translations ${x_i} = \delta {\text{ }}\left( {i - \frac{{k + 1}}{2}} \right), i = 1, \ldots ,k$, of the pattern with a shift amplitude $\delta$, one can obtain the number of collected photons as a function of the relative position with respect to the illumination ${\phi _0}$:

$$\begin{equation*}N = \frac{A}{2}\left( {k + m{\text{ }}\cos \left( {\frac{{2\pi }}{\lambda }{\text{ }}{\phi _0}} \right){\text{ }}\csc \left( {\frac{\pi }{\lambda } \delta } \right){\text{ sin}}\left( {\frac{{k \pi }}{\lambda } \delta } \right)} \right).\end{equation*}$$

For $k = 3$, the modulation efficiency can be analytically expressed as:

$$\begin{equation*}{\varepsilon _m} = \left| {\sin \left( {\frac{\pi }{\lambda } \delta } \right)} \right| m\sqrt {\frac{2}{3}} \frac{{\sqrt {2 + \cos \left( {\frac{{2\pi }}{\lambda } \delta } \right) - \left( {1 + 2{\text{ cos}}\left( {\frac{{2\pi }}{\lambda } \delta } \right)} \right)\cos \left( {\frac{{4\pi }}{\lambda } {\phi _0}} \right)} }}{{3 + m\left( {1 + 2\cos \left( {\frac{{2\pi }}{\lambda } \delta } \right)} \right)\cos \left( {\frac{{2\pi }}{\lambda } {\phi _0}} \right)}}.\end{equation*}$$

As previously stated, a spatial dependence arises for $\delta \ne \frac{\lambda }{k}$. The effect of varying the pattern shift's size onto the modulation efficiency is shown in figure 4. If the modulation efficiency is negligible—the illumination intensity does not change between frames—meLM cannot provide extra information compared to SMLM.

Zoom In Zoom Out Reset image size

Following the same principle as described in figure 3, the triplet of emitter intensities, recorded close to the illumination minimum, is fitted with the illumination profile to obtain the particle location. Comparing figure 3(a) with figure 5(a) highlights the resulting higher localization precision at the same number of photons ($N = 450$) set in this simulation (see specifically the standard deviation of the fitted phase ${\sigma _\phi }$). Reducing the pattern shift will increase the localization precision gain regardless of the number of photons (see figure 5(b)).

Zoom In Zoom Out Reset image size

When we consider emitters residing outside of the illumination minimum, the effect of having a gain in localization precision compared to SMLM will be lost. The farther the particle climbs up towards the peak of the illumination pattern, the less of an advantage is created by the smaller phase shift mode, eventually removing all information encoded in the phase. The observed localization precision gain with respect to the position of the emitter relative to the illumination pattern is shown in figure 6(a). Balzarotti et al showed that the localization precision scales with $L/\sqrt N$ when a single molecule emitter is scanned with pattern shifts of size $L$. However, as illustrated in figure 6(b), this comparison for a fixed number of detected photons implies a strong increase of the fluorescence whenever the fluorophore is located outside of the illumination minimum. Specifically, to keep the same number of photons, an increasing laser power is necessary, thereby creating a larger amplitude of the illumination pattern (see figure 6(b)). If on the other hand the emitter resides close to the peak of the illumination pattern, the localization precision will be low, and the high laser power will eventually bleach the fluorophore. To quantify this increased intensity required, we represent the achievable localization gain alongside the square root of the relative illumination amplitude (figure 6(c)).

Zoom In Zoom Out Reset image size

3.3. Experimental results

In order to demonstrate experimentally the results obtained through computer simulations, a variety of samples have been tested by different research groups. We show in figure 7 a comparison of localization precision between classical SMLM techniques (ThunderSTORM [55]) and SIMPLE. This comparison is carried out on single Alexa488 dyes on glass, both in regular and high-gain regimes, validating a 2 and 6.5-fold improvement in localization precision (figures 7(b)–(d)). The illumination pattern and the fitting procedure effectively provide a doubling in localization precision over the whole field of view, whereas smaller pattern translation give rise to higher gain in regions restricted to the trough of the illumination, similarly to MINFLUX. The datasets are acquired on a modified custom-built TIRF-SIM setup with a high NA objective (NA 1.49).

Zoom In Zoom Out Reset image size

All the meLM techniques require the precise measurement of the induced intensity changes in the fluorescence as well as the generation and translation of a well-defined illumination pattern. A range of recently introduced meLM techniques each feature slightly different implementations achieving those goals, and are summarized thereafter.

4.1. Illumination

4.2. Detection

Currently, the original MINFLUX implementation of meLM still achieves the highest spatial resolution with minimal photobleaching. This implementation based on a doughnut-shaped excitation pattern readily enables a variety of modalities, such as nanoscale imaging and tracking of single fluorophores. The parallelization of the meLM scheme has the advantage of increasing the achievable spatial resolution relative to SMLM, which is the current 'gold standard' for super-resolution microscopy, based on its simplicity and low cost of components to realize it. With the 'simple' addition of a well-characterized illumination pattern and by acquiring a few more images relative to SMLM, the wide-field implementations of meLM—SIMPLE, ROSE, SIMFLUX, and ModLoc—provide enhanced precision over large FOVs, making these approaches very powerful. However, to enable the high-gain capabilities across the entire field of view, allowing virtually unlimited resolution, the use of photo-switchable probes is required.

For practical purposes, a combination of both an initial localization of molecules, by peak finding or SMLM, followed by a meLM procedure, will likely be necessary in order to utilize the full potential of this new class of techniques. By combining meLM with machine learning to allow high-throughput data analysis [42–45] one can anticipate further improvements in quality and applicability of meLM for future life-science research. This should then enable true wide-field imaging of a large number of structures with extensions of less than 10 nm, which will open up new perspectives in the characterization of local macromolecular interactions within single cells. This could be particularly powerful in our on-going quest of unraveling the nanoscale structure of interphase chromatin. Based on these exciting prospects, we believe that it is well deserved to call meLM, in its various implementations, the fourth super-resolution method.

L R acknowledges support of a fellowship from 'la Caixa' Foundation (ID 100010434, LCF/BQ/IN18/11660032) and funding from the European Union´s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 713673. V R acknowledges support from the Spanish Ministry of Economy and Competitiveness through the Program 'Centro de Excelencia Severo Ochoa 2013-2017', the CERCA Programme/Generalitat de Catalunya, MINECO's Plan Nacional (BFU2017-86296-P) and support from the CRG Advanced Light Microscopy Facility. S W acknowledges support from the Spanish Ministry of Economy and Competitiveness through the 'Severo Ochoa' program for Centres of Excellence in R&D (SEV-2015-0522), from Fundació Privada Cellex, and from Generalitat de Catalunya through the CERCA program.