High-cadence Imaging and Imaging Spectroscopy at the GREGOR Solar Telescope—A Collaborative Research Environment for High-resolution Solar Physics

The 1.5 m GREGOR solar telescope is the largest telescope in Europe for high-resolution solar observations (see Soltau et al. 2012 for the origin of the telescope's name and why it is formatted in capital letters). Located at Observatorio del Teide, Izaña, Tenerife, Spain, the telescope exploits the excellent and stable seeing conditions of a mountain-island observatory site. The concept for the telescope's mechanical structure (Volkmer et al. 2012) and the open design employing a foldable-tent dome (Hammerschlag et al. 2012) allow for wind flushing of the telescope platform, which minimizes dome and telescope seeing. The GREGOR telescope uses a double Gregory configuration (Soltau et al. 2012) to limit the field of view (FOV) to a diameter of 150'' to facilitate on-axis polarimetric calibrations in the symmetric light path and provide a suitable f-ratio to the GREGOR Adaptive Optics System (GAOS; Berkefeld et al. 2012) and the four post-focus instruments: the Broad-Band Imager (BBI; von der Lühe et al. 2012), GREGOR Infrared Spectrograph (GRIS; Collados et al. 2012), GFPI (Denker et al. 2010; Puschmann et al. 2012 and references therein), and HiFI (C. Denker et al. 2018, in preparation), which replaced the blue imaging channel (BIC) that used much slower and outdated facility cameras. In the present study, we discuss data processing, analysis, management, and archiving for the last two instruments.

The GFPI is a tunable, dual-etalon imaging spectrometer, where the etalons are placed near a conjugated pupil plane in the collimated beam. The etalons were manufactured by IC Optical Systems (ICOS) in Beckenham, UK. They have a 70 mm diameter free aperture and possess both high finesse (${{ \mathcal F }}_{\mathrm{eff}}=45\mbox{--}50$) and high reflectivity (R ≈ 95%). Their coatings are optimized for the spectral range 5300–8600 Å, where the instrument achieves a spectral resolution of ${ \mathcal R }\approx 250,000$. Spectral scans are recorded with two precisely synchronized Imager QE CCD cameras from LaVision in Göttingen, Germany. One camera acquires narrowband filtergrams, whereas the simultaneous broadband images enable image restoration of the full spectral scan using various deconvolution techniques.

The 12 bit images with 1376 × 1040 pixels are captured at a rate of up to 10 Hz for full frames, depending on exposure time. The image scale of about 004 pixel⁻¹ yields an FOV of 55'' × 42''. The image acquisition rate can be doubled with 2 × 2 pixel binning. The relatively small full-well capacity of 18,000 e⁻ and low readout noise of the detectors are well adapted to the instrument design, considering the small full width at half maximum (FWHM) of the double-etalon spectrometer of just 25–40 mÅ, the maximum quantum efficiency of 60% at 5500 Å, and the short exposure times (t_exp = 10–30 ms) needed to "freeze" the seeing-induced wavefront distortions. The typical cadence of Δt = 20–60 s for a spectral scan provides very good temporal resolution so that dynamic processes in the solar photosphere and chromosphere can be resolved.

However, both exposure time and cadence are already compromises because the high spectral resolution leads to a low number of incident photons at the detector, and small-scale features potentially move or evolve significantly in the given time interval. Faster detectors combining low readout noise, comparatively small full-well capacity, good quantum efficiency, and a high duty cycle with respect to the exposure time mitigate these limitations and are considered for future upgrades. In principle, the GFPI can be operated in a polarimetric mode, which produces spectral scans of the four Stokes parameters so that the magnetic field vector can be inferred for each pixel. However, validation of the polarimetric mode is still in progress. Consequently, imaging polarimetry is not covered in this study.

Sample data products obtained from a scan of the strong chromospheric absorption line Hα 6562.8 Å are compiled in Figure 1 to illustrate the GFPI's science capabilities. Ellerman bombs (see Rutten et al. 2013 for a review), as opposed to microflares, are typically observed as enhanced line-wing emission in Hα, Hβ, Hγ, etc. and have typical lifetimes of 1.5–7 minutes with a maximum of around 30 minutes (Pariat et al. 2007). To resolve their evolution requires fast spectral scans with imaging spectroscopy and ex post facto image restoration to uncover small-scale dynamics within the photosphere and chromosphere. On 2014 August 14, active region NOAA 12139 was observed with the GFPI focusing on a complex group of sunspots and pores (see broadband image in Figure 1). Image restoration using Multi-Object Multi-Frame Blind Deconvolution (MOMFBD; Löfdahl 2002; van Noort et al. 2005) was applied to the spectral data to enhance solar fine structure. The blue and red line-wing filtergrams reveal two features with different spectral characteristics: an Ellerman bomb (Ellerman 1917) as localized, small-scale brightenings (area "a") and a system of dark fibrils with strong downflows associated with newly emerging flux (area "b"), respectively.

Zoom In Zoom Out Reset image size

High spatial resolution imaging and imaging spectroscopy belong to the standard observational techniques of large-aperture solar telescopes. Various studies based on GFPI data illustrate the potential of the instrument. Recently, Kuckein et al. (2017b) used GFPI Ca ii 8542.1 Å filtergrams to study sudden chromospheric small-scale brightenings. The combination of ground-based, high-resolution imaging spectroscopy and synoptic EUV full-disk images from space reveals that the brightenings belong to the footpoints of a microflare. To further investigate the bright kernels and the central absorption part (below the flaring arches), spectral inversions of the near-infrared Ca ii line are performed with the NICOLE code (Socas-Navarro et al. 2015). The retrieved average temperatures reveal rapid heating at the brightenings (footpoints) of the microflare of about 600 K. The inferred line-of-sight (LOS) velocities at the central absorption area show upflows of about −2 km s⁻¹. In contrast, downflows dominate at the other footpoints.

In another study, Verma et al. (2018) used high-resolution imaging and spectroscopic GFPI data in the photospheric Fe i 6173.3 Å spectral line to infer the three-dimensional velocity field associated with a decaying sunspot penumbra (Figure 2). The velocities in the decaying penumbral region deviate from the usual penumbral flow pattern because of flux emergence in the vicinity of the sunspot. The detailed analysis is based not only on GFPI data but also HiFI and GRIS observations, which provide further photospheric and chromospheric diagnostics. In a study with a similar setup for GREGOR multiwavelength and multi-instrument observations, Felipe et al. (2017) investigated the impact of flare-ejected plasma on sunspot fine structure, i.e., a strong light bridge, which experienced localized heating and changes of its magnetic field structure.

Zoom In Zoom Out Reset image size

Detectors based on scientific complementary metal-oxide-semiconductor (sCMOS) technology have become an alternative to standard CCD devices in astronomical applications, in particular when high-cadence and large-format image sequences are needed. In early 2016, we installed HiFI at the GREGOR solar telescope, where it observes the blue part of the spectrum (3850–5300 Å) using a dichroic beam splitter in the GFPI's optical path. Two Imager sCMOS cameras from LaVision (LaVision 2015) are synchronized by a programmable timing unit and record time series suitable for image restoration either separately for each channel or by making use of both channels at the same time.

In typical observations, two of three spectral regions are selected, i.e., Ca ii H 3968.0 Å, Fraunhofer G-band 4307.0 Å, and blue continuum 4505.5 Å (see Figure 3). The width of the filters is around 10 Å, so typical exposure times reach from a fraction of a millisecond to a few milliseconds. Thus, observations are not "photon-starved," as often encountered in imaging spectropolarimetry, where the transmission profile of (multiple) Fabry–Pérot etalons can be as narrow as 25 mÅ. Different count rates in both channels are balanced by choosing suitable neutral density filters. Short exposure times are essential to freeze the wavefront aberrations in a single exposure. The 2560 × 2160 pixel images with an FOV of 648 × 546 are digitized as 16 bit integers and recorded with a data acquisition rate of almost 50 Hz. Thus, the image scale is about 0025 pixel⁻¹, or about 18 km on the solar surface at disk center. The diffraction-limited resolution of the GREGOR telescope with a diameter D = 1.5 m at a wavelength λ = 4000 Å is α = λ/ D = 0055. According to the Nyquist sampling theorem, HiFI images are critically sampled at the shortest wavelength of the standard interference filters.

Zoom In Zoom Out Reset image size

In the standard HiFI observing mode, sets of 500 images are captured in each channel in 10 s and continuously written to a RAID-0 array of SSDs at a cadence below 20 s. The imaging system has achieved a sustained write speed summed over both channels of up to 660 MB s⁻¹. To limit the final data storage requirements, only the best 2 × 100 images of a set are kept for image restoration and further data analysis. These settings are already a compromise, considering that solar features move with velocities of several kilometers per second in the photosphere and several tens of kilometers per second in the chromosphere, often exceeding the speed of sound. Eruptive phenomena in the chromosphere reach even higher velocities, in excess of 100 km s⁻¹. Thus, in the standard observing mode, solar features moving at more than 2 km s⁻¹ and traversing a pixel in less than about 10 s will be blurred, and their proper motions will not be properly resolved. However, considering that HiFI provides mainly context data and only some pixels are exposed to high velocities, this compromise is acceptable. If higher temporal resolution is required—for example, when tracking small-scale bright points or following the evolution of explosive events—changing the observing mode is an option, i.e., reading out smaller subfields to increase the data acquisition rate or dropping frame selection to keep all observed frames.

As demonstrated above, many instrument designs in solar physics rely on high-cadence imaging. The data challenge arises from the combination of several factors: the daytime correlation timescale of the seeing, the evolution timescale of the solar features, and large-format detectors. The latter are needed to either catch transient events or observe large-scale coherent features that provide structuring and connectivity in the solar atmosphere. To illustrate the implications, we carried out a small experiment with the HiFI cameras. Images were acquired at 135 Hz for only a small FOV of 640 × 640 pixels, i.e., 163 × 163 on the solar disk. Three sets of 50,000 images were written to disk with short interruptions at a sustained rate of 220 MB s⁻¹. An extended study based on an even higher image acquisition rate is presented in Denker et al. (2018), where we evaluated image-quality metrics and the impact of frame selection for AO-corrected images on image restoration with the speckle-masking technique (Weigelt & Wirnitzer 1983; de Boer 1993; von der Lühe 1993).

The median filter gradient similarity (MFGS; Deng et al. 2015) is an image-quality metric recently introduced into solar physics. The results for G-band images of the full time series are depicted in the top panel of Figure 4, whereas the two lower panels show successively shorter time periods centered around the moment of best seeing conditions. The AO system locked on a decaying pore in active region NOAA 16643 that contained a light bridge, umbral dots, and indications of penumbra-like small-scale features, i.e., elongated, alternating bright and dark features at the umbra–granulation boundary, as well as chains of filigree in the neighboring quiet Sun that point radially away from the pore's center (Figure 5). In the standard HiFI observing mode, 2 × 100 images are selected within a 10 s period, as indicated by the gray vertical bars in the lower right panel of Figure 4. This observing mode relies on a special implementation of frame selection (Scharmer 1989; Scharmer & Löfdahl 1991; Kitai et al. 1997), where not just the best solar image but a set of high-quality images is chosen for image restoration.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Obviously, even at this very high cadence, the image-quality metrics still show strong variations, despite some clustering of the best images in Figure 4, indicating that seeing fluctuations occur at even higher frequencies. Considering what lies ahead for high-resolution imaging, the data acquisition rates for the next generation of 4k × 4k pixel detectors recording at 100 Hz will amount to about 3 GB s⁻¹. This exceeds today's typical data acquisition rates of less than 1 GB s⁻¹ but demonstrates that the data challenge will persist for years to come and, in particular, for the next generation of large-aperture solar telescopes, such as DKIST and EST. In Denker et al. (2018), we demonstrated that an image acquisition rate of f_acq = 50 Hz is an appropriate choice, considering the marginal benefits in quality for the frame-selected images with respect to the demands on camera detectors, network bandwidth, data storage, and computing power.

Multi-frame blind deconvolution (MFBD; Löfdahl 2002) is another commonly used image-restoration technique in solar physics. Figure 5 compiles some results based on the best set of G-band images. In Denker et al. (2018), we established a benchmark for G-band images, i.e., an MFGS value of m = 0.65, where image restoration with the speckle-masking technique becomes possible. Thus, the seeing conditions were only moderate-to-good on 2017 March 25. The aforementioned threshold was determined for the full FOV of the sCMOS detector, whereas the present observations used a much smaller FOV covering the immediate neighborhood of the AO lock point. The top row of Figure 5 demonstrates that, even compared to telescopes with 0.7–1 m apertures, data obtained with larger 1.5 m class telescopes require a significant increase in the number of Zernike modes (n_mod ≈ 300) in the restoration process. This significantly increases the computation time and poses challenges for imaging with 4 m class solar telescopes. MFBD has an advantage over speckle masking because it requires a smaller number of images for a restoration, in particular, when the images are of high quality taken under very good or excellent seeing conditions. Thus, restored time series with higher cadence become possible. Already, n_sel = 20 selected images deliver good restorations. Note that MFGS loses its discriminatory power for restored images, so that other metrics, like the rms-intensity contrast c, become more important. A likely explanation is that the MFGS metric is sensitive to the fine-structure contents of an image, which is mainly encoded in the phases of the Fourier-transformed image. Thus, once almost diffraction-limited information is recovered, the MFGS metric reaches a plateau. The image contrast, on the other hand, is more closely related to the Fourier amplitudes.

High-resolution solar observations are taken at 1 m class ground-based solar telescopes around the world and with Hinode/SOT from space, where the latter provides, in many respects (i.e., imaging and spectropolarimetry), the closest match to HiFI and GFPI data. Consequently, scientists who work with such high-resolution data are among the primary target groups for utilizing GFPI and HiFI data. In addition, we especially foresee close interactions with the solar physics community on spectral inversion codes and numerical modeling of the highly dynamic processes on the Sun. Therefore, our immediate priority is providing a CRE fostering collaborations among researchers with common interests. Our goal is to raise awareness of and stimulate interest in complex and heterogeneous spectropolarimetric data sets, which are inherent trademarks of imaging spectropolarimeters with a broad spectrum of user-defined observing sequences. Offering a data repository to the whole solar physics community or the general public at large will enhance the impact of these high-resolution data products. However, in this case, it may be advantageous to highlight key data sets obtained under the best seeing conditions or of particularly interesting events like solar flares. In any case, data access, as described in the data policy (Section 4.2), is granted to everyone who registers for the GREGOR GFPI and HiFI data archive. At a later stage, when the high-level data products have matured, a more differentiated access will be instantiated, dropping the registration requirement for the publicly available data.

The data-specific challenges for GFPI and HiFI are as follows.

• 1.  
  The campaign- and PI-oriented nature of the data, which results in different combinations of post-focus instruments with changing setup parameters, e.g., selection of diverse spectral lines, dissimilar spectral and spatial sampling, and different cadences.
• 2.  
  Observations through Earth's turbulent atmosphere, which brings about data with fast-changing quality and necessitates image restoration, where different restoration algorithms introduce multiplicity in the major data levels.
• 3.  
  The complexity of the data sets, which requires considerable efforts to condition the data products for a broader user base—even at the level of quick-look data.
• 4.  
  The availability of personnel and financial resources for maintaining long-term access to data and for quality assurance beyond the typical funding cycle of third-party financial support.

Fortunately, AIP's mission includes the development of research technology and e-infrastructure as a strategic goal. Thus, the collaboration between e-science and solar physics allowed us to develop a tightly matched solution to the aforementioned data-specific challenges.

The GREGOR consortium (i.e., the Kiepenheuer Institute for Solar Physics, Max Planck Institute for Solar System Research, and Leibniz Institute for Astrophysics Potsdam) and GREGOR partners (i.e., the Instituto de Astrofísica de Canarias and Astronomical Institute of the Academy of Sciences of the Czech Republic) agreed that, in principle, all data shall be publicly available. In the following, we use the term "consortium" when referring to GREGOR members and partners.

Since observing proposals contain proprietary information and original ideas of PIs and their teams, the data of PI-led observing campaigns will be embargoed for 1 yr. The embargo period can be extended upon request for another year when observations are related to PhD theses. Quick-look data are publicly available after storage at the GREGOR GFPI and HiFI archive and subsequent processing, typically after 4–6 weeks, and these data are not subject to the embargo period. This way, the PI of an observing campaign can be contacted to inquire about potential collaborations even within the embargo period. The consortium encourages, but does not require, collaboration with the originator of the data. All work based on GREGOR data is required to include an acknowledgment (see below), and the consortium asks the authors to include appropriate citations to the GREGOR reference articles published in 2012 as a special issue of Astronomische Nachrichten (Vol. 333/9). A detailed version of the data policy is publicly available on the GREGOR website at AIP.

The archive is based on the Daiquiri¹⁷ framework, which was developed by AIP's R&D group "Supercomputing and E-Science." Daiquiri is designed for creating highly customized web applications for data publication in astronomy. The features of Daiquiri comprise rich tools for user management, an SQL query interface to enable users to directly enter database queries via the website, means to download the results of queries in different formats following standards of the International Virtual Observatory Alliance¹⁸ (IVOA; Quinn et al. 2004), and plotting functions. The interaction with Daiquiri can be scripted using its VO universal worker service (UWS; Harrison & Rixon 2016) interface.

For the GREGOR archive, we use Daiquiri's user management to implement a custom user-registration workflow. New users first register on the GREGOR data portal, and they do not have to belong to the GREGOR consortium. Therefore, registration is possible for anyone with an interest in GREGOR high-resolution data. To emphasize the intended collaborative nature of this research infrastructure, in the following, we use the terms "GREGOR collaboration" or "collaborators" collectively for all registered users. After registration, one of the project managers confirms the new user, again through the GREGOR portal. Only after this organizational confirmation takes place will members of the technical staff activate the user. This procedure is necessary because users receive CRE privileges allowing them processing data on institute-owned computers and editing of websites and blog entries.

Along with their accounts for the web portal, users also obtain Secure Shell (SSH) login credentials for the data access node (Section 4.4). The SSH protocol facilitates efficient and fast distribution of the data products while ensuring modern security and maintainability (in particular, firewall configuration) when compared to the popular but insecure File Transfer Protocol (FTP). On the data access nodes, an elaborate permissions system ensures access restrictions and data security. This is implemented using Linux Access Control Lists (ACLs), which extend the usual file permissions (user/group/world) common on UNIX systems. Users are organized in groups that gain write permissions to certain subdirectories of the GREGOR archive. These ACL directories, which typically comprise data for a specific instrument, data-processing level, and observing day, can have multiple groups, allowing for fine-grained access by the registered users (e.g., when accessing embargoed data), as well as the GREGOR archive administrators.

Besides the user-registration workflow, Daiquiri is also used to set up the GREGOR web pages, some with access restrictions, some public. This includes the data products generated by the sTools data-processing pipeline (see Section 3). Access to the data was initially limited to the GREGOR consortium but is now open to all registered users of the GREGOR GFPI and HiFI data archive. The data sets generated by sTools are currently in the process of being converted to FITS files containing image extensions. This reduces the level of complexity, as metadata are available, and many images of a spectral scan or in a time series are already aggregated. Once converted to FITS format, these data will be integrated into an SQL database and can be accessed using Daiquiri's query functionality.

The large number and high resolution of images and spectra, as well as the computational effort for their postprocessing, demands capable and efficient structures for storage and data management. To make the best use of GREGOR data and to encourage their usage, we implemented a dedicated CRE at AIP. This research infrastructure acted initially as a central hub for the storage and processing of different data products, as well as their distribution within the GREGOR consortium, but it is now open to all interested scientists. The CRE provides data space and data access with different levels of authorization, in addition to computational resources and customized tools for analysis and processing. Participation in the CRE is managed by the GREGOR consortium lead for GFPI and HiFI. Finally, collaborators have the option via the CRE to publish selected and curated "science-ready" data for the solar community, including a minted DOI registered with DataCite.¹⁹

Over the last decade, AIP provided similar CREs for several projects. For collaborations working on simulations of cosmological structure formation, such as Constrained Local UniversE Simulations (CLUES; Gottlöber et al. 2010) and MultiDark (Riebe et al. 2013), AIP hosts hundreds of TB of file storage and a relational database with about 100 TB of carefully curated particle information, halo catalogs, and results of semi-analytical galaxy models. These data products are available via the CosmoSim²⁰ database provided by the E-Science group at AIP. However, observational collaborations, e.g., the Multi Unit Spectroscopic Explorer²¹ (MUSE; Bacon et al. 2010; Weilbacher et al. 2014) collaboration and the Radial Velocity Experiment²² (RAVE; Steinmetz et al. 2006) survey, also rely on a CRE for data management and processing.

The CRE hardware is integrated into the Almagest cluster at AIP. The whole cluster consists of over 50 nodes and about 3 PB of raw disk space. The different machines are connected through a high-performance InfiniBand network. Throughout the cluster, we use the Linux distribution CentOS as an operating system. Directly allocated for the GREGOR CRE are the following.

• 1.  
  One shared compute node acting as a login node and to share the data among the collaboration. Collaborators can log in to this machine using the SSH protocol and copy data to their workstations for further processing and scientific analysis.
• 2.  
  Two storage nodes with 80 TB of storage space reserved for GREGOR data. Each of the nodes contains 24 hard disks, which are combined into one logical RAID volume using a ZFS²³ file system. Using ZFS's send-receive feature, the disk content of the two nodes is mirrored, and they are physically located in different buildings to minimize the risk of data loss due to catastrophic events.
• 3.  
  A dedicated compute node, hosting the sTools pipeline, which is directly connected to the archive. This machine allows users to run their own programs and to visualize GFPI and HiFI (raw) data. This computer is also used for generating and updating the web pages with quick-look data. Remote users can log in by SSH and use virtual network computing (VNC) for a desktop-like environment.
• 4.  
  Additional resources are supplied to host the web applications for the data archive, connecting to the internet, and backups.
• 5.  
  Two internal compute nodes with mirrored installations of the sTools data-processing pipeline. One of the nodes with a 64-core processor and 256 GB RAM is dedicated to image restoration and hosts the SVN repository of the sTools code, including external libraries. The workstations of the researchers on the AIP campus can NFS-mount the respective volumes containing data and software libraries so that data processing and analysis are also possible locally.
• 6.  
  In addition, a copy of the sTools data-processing pipeline is installed at the German solar telescopes at Observatorio del Teide, Izaña, Spain. The computer network includes workstations and a dedicated multicore computer for data processing and image restoration on site. In particular, the MFGS-based image selection for the HiFI data is carried out immediately after the observations with an easy-to-use GUI written in IDL as an interface to sTools.

We distinguish three major levels of data products within the GREGOR GFPI and HiFI data archive.

• 1.  
  Level 0 refers to raw data acquired with GFPI and HiFI. The data are written in a format native to the DaVis software of LaVision, which runs both instruments. A short ASCII header declares the basic properties of the images (e.g., DaVis version number, image size in pixels, number of image buffers, type of image compression, if applicable, etc.), which is followed by either compressed or uncompressed binary data blocks. Another free-format ASCII header, containing auxiliary information like a time stamp with microsecond accuracy, is placed after the data blocks at the end of the file. This time stamp results from the programmable timing unit (PTU), which provides external trigger signals for image acquisition. Specific settings of the observing mode and instrument parameters are saved in additional text files for each image sequence or spectral scan. These set files include, for example, telescope and AO status, as well as a separate time stamp for the observing time based on the camera computer's internal clock, which is synchronized with a local GPS receiver at the observatory.
• 2.  
  The huge amount of large-format, high-cadence HiFI data (up to 4 TB day^–1 with the current setup) requires reducing the data already on site, directly after the observations. This is the standard procedure and includes dark and flat-field corrections. In addition, the image quality is determined with the MFGS metric, which facilitates frame selection. Only the best 100 out of 500 images in a set are kept. The calibrated image sequences (level 1) of the two synchronized sCMOS cameras are written as FITS files with image extensions. The metadata contained in the primary and image headers are partially SOLARNET-compliant (level 0.5), which means that they do not contain all mandatory SOLARNET keywords and have not used any SOLARNET/FITS standard keywords in a way that is in conflict with their definitions. HiFI level 0 data are typically deleted once the frame-selected and calibrated level 1 data are safely stored in the GREGOR archive. HiFI level 0 images are only kept when the seeing conditions were excellent, fast or transient events were captured, or special observing programs were carried out (see Denker et al. 2018). The processing time for restoring a single HiFI image (level 2) from a set of 100 images with KISIP takes several tens of minutes on the 64-core compute node.
• 3.  
  Level 1 GFPI data are typically created at AIP after the observing campaign, which reflects the high complexity of data from imaging spectroscopy. In most cases, the multiple images per wavelength point are simply destretched and coadded with no image restoration. However, all other calibration steps (e.g., alignment of narrowband and broadband images, blueshift and prefilter-curve corrections) are carried out so that scientific exploitation of level 1 data is possible. After preprocessing level 1 data, i.e., after dark and flat-field corrections and determining the alignment of narrowband and broadband images, a copy of the narrowband and broadband images is saved that serves as the starting point for image restoration (level 2), thus avoiding preprocessing level 0 data twice. Only the best spectral scans are then chosen for level 2 processing, i.e., image restoration with MOMFBD or speckle deconvolution. The processing time of a single scan with MOMFBD takes several hours on the 64-core compute node.
• 4.  
  Level 1 data are the starting point for creating quick-look data products such as time-lapse movies, Doppler velocity maps, and overview graphics for seeing conditions and observing parameters.
• 5.  
  Level 2 data are restored data using MOMFBD and KISIP (see Section 3). These data-processing steps are only included on demand, considering the significant amount of computational resources. The best image restoration scheme is chosen by the researcher working with the data, and it is not unusual to select different ways to restore images or spectral scans depending on user preferences or a specific science case. Once the spectral scans are restored, other calibration steps still need to be applied, such as blueshift and prefilter-curve corrections, before physical parameters such as Doppler velocities or other spectral-line properties can be determined.

Level 1 HiFI data (and occasionally level 0) and level 0 GFPI data are transferred from the GREGOR telescope to AIP by regular 2.5 inch external hard-disk drives with 2–4 TB storage capacity. Smaller data sets are often transferred over the internet. However, the physical transfer using hard disks is preferred over copying over the internet due to limited bandwidth and network reliability at the observatory site. Data from all GREGOR instruments can be stored on site on a 100 TB storage array for up to 3 months. Keeping at least two copies of the data at different locations during the data transfer to the GREGOR archive mitigates potential data loss. Data integrity during the transfer is assured by monitoring the transfer logs and verifying that all files with correct sizes were transferred. The total amount of data that was acquired with GFPI and HiFI, as well as with the now-obsolete facility cameras of BIC, is summarized in the last row of Table 1. The other entries in the rows for 2014–2016 refer to the number of scientific data sets that were obtained in various observing campaigns. The data volume refers to the sum over all data levels. GFPI level 1 data are roughly twice the size of level 0 data. Furthermore, the bulk of the data volume arises from level 0 data for BIC and level 1 data for HiFI.

Finally, 70 users are currently registered in the CRE and are mainly from the GREGOR consortium. However, increasingly, external users (at the moment, 10) have registered with the CRE, most of them participating in (coordinated) observing campaigns or the SOLARNET Access Program promoting observing campaigns with Europe's telescopes and instruments for high-resolution solar physics. In the meantime, routine observation started at GREGOR in 2016, and we already see an influx of new international collaborators who will certainly broaden the user base of the GREGOR CRE.

The GREGOR early science phase took place in 2014 and 2015, where members of the GREGOR consortium collaboratively carried out observing campaigns, i.e., in 2014 with the individual instruments and in 2015 with multi-instrument setups. Notably, a 50 day observing campaign with GFPI and BIC (the predecessor of HiFI) was carried out in 2014 July and August. In a joint effort, scientists from all involved institutes submitted observing proposals, which were evaluated and condensed into a list of top-priority solar targets and feasible observing modes and strategies. The observations were carried out by experienced observers of all institutes together with novice observers (not necessarily novice scientists) to strengthen their observing skills and familiarize them with the new instruments. The data, which were acquired with many different setups, were used to develop, test, and improve the sTools data-processing pipeline. Finally, the level 1 data were stored in the GREGOR GFPI and HiFI data archive so that all scientists were able to access and analyze them. Collaborations on specific studies, with the aim of publishing them in scientific journals, were coordinated using the blog facilities of the Daiquiri framework. This also includes the organization of GREGOR science meetings at AIP. The News Blog of the GREGOR CRE allowed us to publish poster presentations given at solar physics meetings (e.g., the SOLARNET IV meeting in Lanzarote in 2016) and newly appearing journal articles and conference proceedings to a broader section of the solar physics community and the interested public. The first results of the GREGOR early science phase were published in 2016 in the journals Astronomy & Astrophysics (Vol. 596) and Astronomische Nachrichten (Vol. 337/10), and more up-to-date GFPI and HiFI science publications are referenced in Section 2.2.

In the future, we plan to expand the data archive and data access infrastructure considerably. With a release of the data to the general solar community, new means of data access will be necessary beyond users actively utilizing the CRE to interact with like-minded researchers. As before, the latter type of access will be managed via registration and subsequent confirmation by the collaboration. While collaborators will still be able to retrieve data through SSH connections, this is not suitable for public access by the general scientific community.

Therefore, we will extend the GREGOR archive toward a public data portal, which will offer a full search on all files in the archive (unless they are still embargoed) and downloads using the HTTP protocol, either through the browser or using command line tools. The selection of files will be based on the metadata of the level 1 and 2 data of the GFPI and HiFI instruments. The archive will also allow for SQL queries to create custom result sets based on any scientific criteria. Standards defined and adopted by IVOA, like UWS and Table Access Protocol (TAP), will expedite accessing the data through clients using interoperable application programming interfaces (APIs). The results of the queries of a user and the queries themselves will be stored in a personal database to be retrieved at the user's convenience without repeating the query to the whole GREGOR database.