Position correction in ptychography using hybrid input–output (HIO) and cross-correlation

Ptychography is a form of coherent diffractive imaging; it employs far-field intensity patterns of the object to reconstruct the object. In ptychography, an important limiting factor for the reconstructed image quality is the uncertainty in the probe positions. Here, we propose a new approach which uses the hybrid input–output algorithm and cross-correlation in a way that can correct our estimates of the probe positions. The performance and limitations of the method in the presence of noise, varying overlap, and maximum recoverable error are studied using simulations. A brief comparison with other existing methods is also discussed here.


Introduction
Computationally imaging an object-replacing lenses with algorithms-is becoming extremely popular. Over the years, several iterative algorithms have been developed including the error-reduction algorithm [1], the hybrid input-output (HIO) algorithm [2], the hybrid projection-reflection algorithm [3], the relaxed averaged alternating reflectors (RAAR) algorithm [4]. These methods reconstruct an object from a single intensity pattern. An algorithm called ptychographic iterative engine (PIE) [5] was developed, which uses several intensity patterns for reconstruction and was found to be superior to other existing methods.
In ptychography, an object is scanned by a localised probe in a way that the neighbouring probe positions should overlap with each other; corresponding to these probe positions, the diffraction patterns are captured in the far-field. These diffraction patterns are employed to reconstruct the complex object. The crucial aspect for the success of this method is the overlap between neighbouring probe positions. The optimum overlap has been found to be about 60% [6].
Ptychography has given significant results with visible light experiments [7], whereas the reconstructions have suffered for x-rays and e-beams data-the limitations were due to the inaccurately known initial parameters, e.g. probe positions [8,9]. The illuminating probe function, for instance, should be known accurately. To this end, extended PIE has been introduced to eliminate the requirement of accurately known probe function [10]. Later on, it was found that probe positions for electron ptychography should be known with an accuracy of 50 pm [11], which is difficult to achieve experimentally. Several successful attempts, subsequently, have been made to solve the probe positions correction problem.
For example, one finds in the literature the 'annealing approach' based on 'trial and error', but this is computationally expensive [12]. In yet another study, nonlinear (NL) optimisation with ePIE is used [13]. NL optimisation, however, can not correct with sub-pixel accuracy. To achieve this, one is required to use the cross-correlation (CC) method [11], which has been widely used. Recently, we have proposed a new method based on gradient of intensity patterns [14] that can correct the probe positions with sub-pixel accuracy while being less computationally expensive than the CC method.
In this work, we introduce a novel method for the correction of the probe positions in ptychography [15]. This method combines the well known techniques HIO and cross-correlation in a way that it can also correct probe positions in ptychography. This is possible mainly due to two important properties: • The reconstructed object using single intensity method is indifferent to wrong/shifted support constraint, except, the reconstructed object is shifted laterally as shown in figure 1. • Sufficient overlap between neighbouring probe positions gives information about the relative shift between neighbouring probe positions. If the probe positions were correct, the overlapped part of the object corresponding to neighbouring probe positions will coincide with each other (see figure 2).
The performance of this method with varying parameters and its limitations are presented here. The paper is organised as follows: in section 2, we elaborate on the new method in detail. Section 3 details the simulation results to asses its performance.
In section 4, we discuss the main findings and its limitations. Finally, we present the conclusions in section 5.

The method
In ptychography, an object O r ( ) is scanned by a probe/ aperture P r R j -( ) and the corresponding intensity patterns I u j ( ) are captured in the far-field. Here r and u are the coordinate vectors in the real and the reciprocal space, respectively. R j is the jth probe position vector. If J is the total number of probe positions, j=1, 2, 3 ... J.
From the property of the Fourier transform, translating a wave-field along the x or y axis in real space does not affect the intensity pattern in the far-field. Therefore, for a single-intensity phase retrieval method, the reconstruction of the object will be indifferent to the position of the support constraint except that the reconstructed object will be at a different place in the real space. In figure 1, the exit wave O P r r r y = ( ) ( ) ( ), the intensity pattern in the far-field, and the reconstruction with shifted support constraint P r R -( ) are shown. Here, the reconstruction was performed using HIO. Now, let us assume that the probe positions in ptychography are not accurately known. If we reconstruct the exit wave r j y ( ) defined by the probe P r R j -( ) using HIO and find the part of the object O r j ( ) corresponding to r j y ( ), we will have the correct part of the object at a wrong probe position. This will create a miss-match in the overlapped region between parts of the object O r The proposed algorithm is a sequential combination of ePIE, HIO and cross-correlation. For the kth iteration, the steps of the algorithm are as follows: 1. Perform a few iterations (say l) of ePIE with the estimated object O r k ( ), the probe P r k ( ) and the probe positions R k j j to obtain a better estimate of the object O r k ¢ ( ) and the probe P r k ¢ ( ). 2. Calculate the exit wave corresponding to each probe position j as: where α is a small number. 5. Calculate the overlapped region S r where β is a threshold parameter. Then, calculate the cross-correlation C as: Determine the vector max r r = for which C r ( ) is maximum, and this vector max r should be equal to the relative shift between neighbouring probe positions Step 5 is performed for all the probe positions j=1, 2, 3 ... J sequentially to find the updated probe positions R k j j Note that, j 1 and j do not always correspond to neighbouring probe positions. Therefore, when probe position j 1 is not the neighbour of the probe position j, we replace j 1 by the neighbouring probe position. In figure 3, the shaded regions show the overlapped parts which are used for the probe positions correction. 6. With the updated probe positions R k j j } , obtain the updated object estimate as a weighted average of the k j y given by: where α is a small number, and update the probe function by: ) as a support constraint, the proposed method has a limitation on the probe function: it should be zero outside the known defined area.

Simulations
To assess the performance of the algorithm, we performed simulations using parameters that correspond to a visible light experiment. Let us suppose that the wavelength of the light is 400 nm; l = the focal length of the lens which was used to create the Fourier transform, is f=10 cm; the detector has 512×512 pixels and the detector pixel size is . The detector pixel size in x and y directions are same. Thus, the size of the pixel along the x-axis in object The object used in the simulations has 224×224 pixels. The illuminating probe was created using a pinhole of radius   We calculate the mean error of the estimated probe positions using the following expression: is the correct probe position and X Y R , In figure 5, the plot for the rms error of the probe positions versus the iteration number are shown. Ten simulations were performed with random initial probe position errors varying from [−5, 5] pixels. The solid line shows the mean and the semi-transparent patch is the standard deviation. As can be seen, for all ten simulations, the probe positions converge to the correct values in one iteration of the proposed method. This method can correct the probe positions with integer pixel accuracy.

Varying number of ePIE and HIO iterations
The entire point of ptychography is that the neighbouring exit waves should overlap with each other, whereas here, the exit waves are also updated separately using HIO. One can question on the added value of ptychography if the object is already being reconstructed using HIO. Hence, we performed ten simulations when ePIE was not used at all. The parameters for the simulations were the same as in section 3.1 except the number of PIE iterations l was zero. As can be seen from the figure 6, the algorithm diverges. It can be explained by the inability of HIO to converge to the correct solution. There is an equal probability for HIO to converge to the twin image of the object if the support constraint is centro-symmetric, which is the case for our simulations. Whereas, a few iterations of PIE gives a better initial estimate for HIO to converge to the correct solution.
On the other hand, as we change the number of HIO iterations while keeping the number of PIE iterations constant, the convergence of the proposed method changes. This can be clearly seen from figures 7 and 8. For each plot, ten simulations were performed where the simulations parameters were same as in section 3.1 except the number of HIO iterations were changing. Figures 7 and 8 show the simulation results for 4×4 and 8×8 probe positions, respectively. The higher the number of HIO iterations, the better the convergence of the algorithm is. For 50, 100, and 200 iterations of HIO, the probe position errors converge after one iteration of the algorithm. On comparing the results for the case 4×4 and 8×8, this method is more robust for the 4×4 case than for the 8×8 case, where the area of the scanned object and the relative overlap were the same. This result may be explained by the fact that the probe is smaller for the 8×8 case, leading to a smaller overlapped area. Since this method tries to match the overlapped part of the object to correct the probe positions, the smaller overlapped area influences the results.

Simulations in the presence of noise
We also performed the simulations using the intensity patterns corrupted with Poisson noise. The number of photons per diffraction pattern was varied from 10 6 to 10 10 and the rms probe position errors are shown in figure 9. For each noise level, ten simulations were performed with random initial offsets taken from [−5, 5] pixels for the probe position. Because RAAR is known to outperform HIO in the presence of noise, 100 iterations of RAAR were used instead of HIO. The other parameters for the simulations were the same as mentioned in section 3.1. For the case of 10 8 , 10 9 and 10 10 photons, most of the time the method gives accurate probe positions. Whereas, for number of photons 10 7  , the method starts to show deviations.  Here the simulations parameters are same as in section 3.1 except the number of iterations for PIE are zero, i.e. l=0. Solid line represents the mean and the standard deviation is shown as the semi-transparent patch. Note that, due to wide variation in the reconstructed probe positions, standard deviation is larger than mean. Therefore, some parts of the plot fall below zero.

Effect of the initial error and the overlap
To evaluate the performance of the algorithm with varying introduced initial maximum offset in the probe positions and overlap, the simulations results are shown in figure 10. Ten simulations were performed for each overlap and introduced initial maximum offsets in the probe positions. Solid lines show the mean of the final converged probe positions of ten simulations; the semi-transparent patch is the standard deviation of the same. As can be seen from figure 10, all probe positions for all ten simulations converged to the correct probe positions when the initial introduced maximum offset was 5 and 10 pixels. As expected, the algorithm is robust for higher overlap between adjacent probes. For 85% overlap, this method corrected probe positions with 100% accuracy for 15 and 10 pixels of error in these ten simulations.

Comparison with intensity gradient (IG) method
For the comparison in terms of convergence, we have shown the simulation results for the IG method and the proposed method in figure 11. Ten simulations for each method were performed with randomly varying initial probe positions error. The simulation parameters were same as in section 3.1. Figure 11(a) shows the simulation results when the IG method [14] was used to correct the probe positions. For IG method, the first 15 iterations were performed with PIE. Probe position correction started from 16th iteration. From 35th iteration onwards, probe update (ePIE) started. Figure 11(b) shows the simulation results when the proposed method was used to correct the probe positions.
The proposed method is correcting probe positions within one iteration. One iteration of the proposed method consists of 10 ePIE, 70 HIO, and 1 cross-correlation. Whereas, for the IG method, the probe positions are corrected within 35 iterations with mean error of 0.80. One iteration of the IG method has 1 ePIE and gradient of intensity (equivalent to two Fourier transforms). The proposed method corrects probe positions with integer pixel accuracy; the IG method corrects with subpixel accuracy. A possible suggestion for the proposed method to achieve sub-pixel accuracy is to use matrix multiplication method proposed by Guizar-Sicairos et al [16] as used in CC method to achieve sub-pixel positions correction [11].

Discussion
In this article, we have proposed an alternative method to correct the probe positions in ptychography which is significantly different from the CC method. The CC method performs cross-correlation between the objects corresponding to consecutive iterations for each probe position. Here, we take the cross-correlation between the objects corresponding to the neighbouring probe positions to match the overlapped region.
In the presence of noise, the proposed method, however, is not as robust as the CC method is. There are a few possible explanations for that. In the presence of noise, some parts of the object have high probability to converge to its twin image or stagnate; thus, they lead to wrong probe positions. Furthermore, since we match the overlapped part of the reconstructed objects corresponding to the neighbouring probe positions, the probe position correction depends on the previous probe position. Therefore, one wrong probe position can propagate this error to the other probe positions as well.
All the simulations presented in this article are based on one assumption: the probe function is zero outside the known  area, because the probe function is used as the support constraint while employing HIO. Due to this limitation, it can not be used in every possible application of ptychography. One specific application would be to reconstruct the wavefront, when it is scanned by a mask.
Another important observation is that our method gives better results for a smaller number of probe positions than large number of probe positions, where the scanned area of the object and the relative overlap for the both cases were same. A possible explanation for this is that the size of the probe is smaller for the case of large number of probe positions, which results in an even smaller overlapped area. Since this method tries to match the overlapped region, it does not perform well for a small overlapped area.

Conclusion
We have devised a novel technique which combines HIO and cross-correlation to correct the probe positions in ptychography. This method can correct probe positions with integer pixel accuracy. We have so far not found a case where this method outperforms the IG method or CC method. The results of the study show that it can be used as an alternative method for probe position correction. It has a limitation on the probe function which must be zero outside the defined area of the probe function. Due to this limitation, it can not be used in every possible application of ptychography. We, however, anticipate that this method, for example, can be used for wavefront reconstruction applications. Furthermore, these  findings may help us to understand the probe position problem from a different perspective.