Cyberhubs: Virtual Research Environments for Astronomy

Web-Exploration for NuGrid Data Interactive WENDI provides

• 1.  
  python and bash notebooks, terminal and text editor;
• 2.  
  example notebooks; and
• 3.  
  self-guided GUI notebooks (widgetized notebooks).

The WENDI widget notebook NuGridStarExplorer (Figure 5) provides GUI access to plotting and exploring the NuGrid data sets, specifically the library of stellar evolution and detailed nucleosynthesis simulations. As an example, the evolution of a low-metallicity, intermediate-mass star is shown during the asymptotic giant branch evolution. The Kippenhahn diagram shows the Lagrangian coordinates of recurring He-shell flashes, each of which drives a pulse-driven convective zone. In the pulse-driven convective zone the ${}_{}^{22}\mathrm{Ne}(\alpha ,n{)}_{}^{16}{\rm{O}}$ reaction creates neutrons with neutron densities reaching ${N}_{n}\approx {10}^{12}\,{\mathrm{cm}}^{-3}$. At these neutron densities s-process branchings, such as at ${}_{}^{95}\mathrm{Zr}$, are activated and the neutron-heavy ${}_{}^{96}\mathrm{Zr}$ is produced. Figure 6 shows the isotopic abundance distribution of s-process elements. Note how the mass fraction of ${}_{}^{96}\mathrm{Zr}$ exceeds that of ${}_{}^{94}\mathrm{Zr}$ in the shown model at the end of the pulse-driven convective zone. In the ${}_{}^{13}{\rm{C}}$ pocket, where the bulk of the s-process exposure takes place, ${}_{}^{94}\mathrm{Zr}{/}_{}^{96}\mathrm{Zr}\approx 600$ because the neutron density is low and the branching is closed. In the solar system ${}_{}^{94}\mathrm{Zr}{/}_{}^{96}\mathrm{Zr}=6.2$. This demonstrates the different neutron density regimes of the s process. Although this example is for low metal content, conditions at solar-like Z are similar. The solar system abundance distribution originates from a mix of low- and intermediate-mass stars, involving both the ${}_{}^{13}{\rm{C}}$ and the ${}_{}^{22}\mathrm{Ne}$ neutron source, which each produce isotopic ratios in different proportions (Gallino et al. 1998; Herwig 2005, 2013).

Zoom In Zoom Out Reset image size

While the widget notebooks provide easy and powerful access to the data, customized analysis may require the added control of programming access to the platform. In order to make it easy for users to get started analyzing the NuGrid data, we keep adding to a collection²⁷ of short example analysis tasks, such as abundance profiles at collapse, or C13-pocket analysis. All data and software dependencies of these examples are satisfied on wendihub. Although users can easily clone their own copy of any external repository, the wendi-examples are preloaded in WENDI and are a convenient starting point for further analysis. Users who create an interesting new example are encouraged to fork the example repository on the terminal command line, add their new example, and make a pull request to the original repository. Although executing notebooks is just a matter of clicking the play button, any further interaction for this type of notebook-based analysis requires basic knowledge in Python.

Two additional widget notebooks²⁸ are currently available in WENDI. The OMEGA self-guided interface provides example applications of the NuPyCEE (Ritter & Côté 2016) code and OMEGA, such as models for dwarf galaxies Fornax, Carina, and Sculptor. For OMEGA WENDI allows arbitrary, complex programming of analysis through ipython notebooks as well. For example, the top panel of Figure 7 shows that NuGrid massive star yields overproduce some iron-peak elements, such as Cr and Ni, but produce a consistent amount of other elements, such as Ti, V, Cu, and Ga. Further analysis of the yield source, as shown in the IMF-weighted yields (bottom panel), reveals that the overestimation of Cr in the galaxy evolution models originates from the $20{M}_{\odot }$ model at Z = 0.01. This analysis tool allows us to identify sources of discrepancies between numerical predictions and observations. In this case, further analysis of the underlying stellar evolution model using similar approaches to those demonstrated below for intermediate-mass stars, which will be presented elsewhere in more detail, connects this overproduction of some iron-group elements to the convective merger of an O-Si shell in the stellar evolution model, which may not be realistic. This analysis can be performed by anyone on WENDI and is available there in a notebook as part of the preloaded WENDI examples.²⁹

Zoom In Zoom Out Reset image size

Another widgetized notebook provides an interface for Stellar Yields for Galactic Modeling Applications (SYGMA; Ritter et al. 2017a), which allows us to generate simple stellar population models and retrieve chemical yields among other properties in table formats that can be used as building blocks for GCE models or hydrodynamic simulations of galaxy evolution. The SYGMA section of WENDI also contains a folder with notebooks³⁰ that run the SYGMA code and generate all plots shown in the SYGMA code paper (Ritter et al. 2017a). This serves as an example of how cyberhubs can be a tool in support of the goal of reproducible science, by providing access not only to data and code but also to the capability to execute the analysis on the data in a controlled and specified environment.

A technical detail concerns how the widget notebooks are launched. The cyberhubs configuration allows automatic self-start-up that hides Python code cells, creating a relatively polished final experience. For this to work, the widget notebooks have to be designated as trusted. Jupyter references a database in a notebook signatures database file that contains instances of the notebooks that are configured to be trusted. In order for a notebook to be trusted, the raw .ipynb notebook file must exactly match the file used to sign said notebook in the database. This system implemented by JupyterHub is rather sensitive to minute changes in the notebook and does not easily incorporate notebook updates and preserve the trusted state of the previously signed notebook. In order to ensure flexibility of the application hubs with trusted notebooks in their respective singleuser images, even after a notebook has been updated, a bash script is used to trust the notebooks in the state that they exist when staged. This ensures that even if a notebook has changed remotely prior to building a singleuser application hub with a reference to an outdated signatures database, the database will be updated automatically. This script contains a series of paths to the notebooks in the singleuser environment that the user requires to be trusted. When it is run, it signs the notebook signatures database file in the singleuser notebook directory.

WENDI is provided by the wendihub docker image that can be found on docker hub (cyberhubs/wendihub), as well as in the astrohubs GitHub repository.³¹

The third of the above cases adds the requirement to install and run multicore, parallel simulations. The NuGrid collaboration uses the MESA code that uses OpenMP to provide shared memory parallelism scaling to up to $\approx 10$ cores. Like many sophisticated simulation codes, MESA relies on a significant number of dependencies. The installation processes have been significantly eased with the MESA-SDK. Still, installation can be a challenge, especially for inexperienced users. The NuGrid code NuPPN for single-zone, multizone, and tracer particle processing adopts MPI parallelism and exhibits good strong scaling to $\approx 50$ cores for typical 1D multizone problems. Both applications are compiled with the gfortran compiler and require hdf and se libraries, as well as numerical libraries, such as SuperLU and OpenBLAS. The singleuser application mesahub combines the compilers, libraries, and environment variable settings needed to install and run MESA and NuGrid simulation codes, and probably several other codes with the same requirements. The currently latest mesahub docker image (version 0.9.5) runs the NuGrid codes NuPPN/mppnp for parallel multizone simulations and NuPPN/ppn for single-zone simulations and has been tested for MESA versions 8118, 8845, and 9331. It should be straightforward to update this application to accommodate both newer and older MESA versions. The resources of the entire host machine can be accessed by each user through their application docker container. We are currently running cyberhubs on several servers, including one instance on a virtual workstation with 16 cores and 120 GB memory, which allows several concurrent MESA runs, as well as using all 16 cores for NuPPN multizone simulations.

In addition, this application includes a wide range of Python packages for data analysis, including NuGrid's NuGridPy toolbox. The application also includes common command-line editors and a complete LATEX installation that allows manuscript generation, complemented with the browser's PDF viewer. Jupyter extensions that allow easy generation of interactive slide shows from notebooks for presentations are included. It is therefore possible to perform all steps needed for a research project just inside the mesahub application.

Over many years, the team at the University of Minnesota's Laboratory for Computational Science & Engineering (LCSE) has developed a series of tools to deal with the voluminous data that are generated by collections of large 3D fluid dynamics simulations. The LCSE was formed in 1995, but the activity began in 1985 as a result of the University of Minnesota's purchase that year of the largest and most powerful supercomputer then available, the Cray-2. Only three of these machines existed in the world at that time. This purchase, coupled with the rarity at the time of academic researchers with simulation codes capable of exploiting this machine, produced an unprecedented opportunity. The university had purchased the machine, but not a data storage system. An early way around that problem for our research team was to take advantage of a holiday sale of disk drives by Control Data, and later the purchase of a tape drive and a small computer to drive it. With this experience, a long tradition of ever greater data compression and development of data analysis and visualization tools began. At the time, there were no data file format standards, nor were there tools, aside from programs one could write oneself, to read such files. The result of this combination of circumstances was that, in the LCSE, we developed our own very powerful tools and techniques to analyze and visualize 3D simulation data. As the field has grown, other groups have taken it upon themselves to produce, enhance, and maintain such tools for community use, which is a full-time activity that we chose not to engage in. The infrastructure described in this article has provided a framework in which we can embed our LCSE tools, giving them an interface that can be quickly understood and utilized by others through a web browser. Python is the glue that connects our utilities to the framework and to external users. All this can now make our simulation data available to a community of interested parties around the globe.

Simulations in three dimensions pose special challenges to the understanding of the computational results. LCSE did not embark on 3D simulation until we heard from a Pixar representative in the mid-1980s about the invention of volume rendering. Once we saw on a workstation screen the rotating image of the volume-rendered water rat that constituted the first demo of the technique, it was obvious to us that this was the solution to the data exploration problem for our fluid dynamics domain. Soon thereafter, we had our first volume rendering program, written by David Porter, running on the Cray-2. This new visualization technique (Ofelt et al. 1989; Porter & Woodward 1989) prompted the conversion of our 2D hydrodynamics code to do 3D simulations. Initially, we tried to save as much of our simulation data as we possibly could, because 3D fluid dynamics simulation was so new that we thought that we could not possibly predict what representations of the data we might later wish to make. We quickly discovered that we could compress saved data down from 64 to 16 bits per number, saving a factor of 4 in data volume. Even so, the data volume was enormous. Decades of experience with visualizing and analyzing 3D simulation data (Woodward 1992a, 1992b, 1993; Tucker & Woodward 1993) have led us to the approach described below that is connected to the Python-based cyberhubs framework described in this article.

Our simulations fall into fairly simple categories, such as homogeneous turbulence, stellar convection in slab or spherical geometry, or detailed studies of multifluid interface instability growth in slab or spherical geometry. In each category, we do many simulations that all have common features. This has meant that after the first few simulations are completed, we have a very clear idea of what data we want to preserve and what visualizations we want to make of the flows in any new category. We have also developed highly robust nonlinear maps from the real line, or the positive real line, to the interval from 0 to 255. Each such map is determined by an initial functional transformation, such as a logarithm, for example, followed by the standard nonlinear map given by the values of just two constant parameters. Such mappings of simulation data to the 256 color levels used in volume rendering can work over all the runs in a single category of simulations without any modification. This not only is tremendously convenient but also allows direct and meaningful visual comparisons of data from different runs. It turns out that certain color and opacity maps can work well for a particular simulation variable, such as vorticity magnitude, over an entire category of runs without any modification. These to some degree unexpected findings have profound consequences for data compression.

The robustness of nonlinear mappings from the real line to color levels allows us to have our codes dump out only a single byte per grid cell per variable field. This is an enormous savings in data volume. It can only be helpful if one can know before doing the simulation which variables are useful for visual exploration of the simulation results. One can save even more data volume if one knows which views of such variables one wants to preserve. Such images can then be compressed further by standard image file formats. Making this data compression also requires that one know the color and opacity mapping one wishes to use. After a run is completed, these images can now be animated using standard movie making software, such as mencoder, that have replaced our own movie animation software. However, to save only images and not the much more voluminous raw voxel data, one must build the volume renderer into the simulation code. We have done this using the srend software package (Wetherbee et al. 2015). In this way rendering with srend happens right in the code rather than as a separate activity as part of a complex workflow, which has many benefits. The full-resolution voxel data need never be written to disk at all, although, for now, we still write this out just in case. This new capability replaces the previous workflow involving the LCSE HVR volume renderer that required slow and difficult data format conversion, GPUs, and special software libraries to run. srend is written in Fortran, is compiled along with the simulation code, and has no dependencies on software libraries. Using either the new srend capability or the previous HVR volume renderer, our strategy is to create default image views of a pre-defined set of variables for all dumps. These image libraries are available through the cyberhub.

Even if we were to continue to save full-resolution voxel data, these data sets are very clumsy to work with owing to their sheer size of 45 GB per dump depending on how many variables are saved. At the same time, a flexible capability to make any visualization of any variable after the simulation is highly desirable. We accomplish this by performing an additional data compression. This final data compression has evolved from our use of simulation data to validate and develop statistical models of turbulence (Woodward et al. 2006).

Turbulence closure models deal in averages. They tend to be based on comparisons of averages of products and the products of the corresponding averages. To work with such models in the early 2000s, we used averages of our simulation data taken over cubes 32 cells on a side. Our filter represented the behavior of a quantity inside the filter cube by a quadratic form determined by the 10 lowest-order moments of the quantity. This filtering technique was derived from our work with the PPB advection scheme (Woodward 1986; Woodward et al. 2015), which also works with the 10 lowest moments. To be able to construct such filtered representations using a moving filter volume, from our simulations we saved averages of many different variables taken over cubes 4 grid cells on a side. We saved these with 16 bit precision, after first passing them through our robust nonlinear maps.

Our present simulation codes all now work with fundamental data structures consisting of briquettes of 4 grid cells on a side. The problem domain is subdivided into regions, which are rectangular solids, and the regions are subdivided into grid bricks, which are smaller rectangular solids. Each grid brick is a brick of briquettes, augmented all around by a single layer of ghost briquettes from the 26 neighbor bricks.

For the storage required to save just one byte of data from each grid cell in our simulation at any dump time level, we can instead save, with 16 bit precision, 32 variable averages for each grid briquette of 64 cells. Thirty-two variables is so many that we can save several other quantities that require differentiating the simulation data and are useful for model building, in addition to storing variables that we nearly always look at. We include, for example, the magnitude of the vorticity, the divergence of the velocity, and both volume-weighted and mass-weighted averages of the velocity components. The idea is that from the 32 quantities one could derive almost anything one may need when analyzing the data.

Data cubes with the 32 variables representing each of the 216 or 512 regions of a large simulation are saved directly by the code as it runs in separate disk files. In this way the analytic tools have immediate and targeted access to any desired region. In practice, it has been somewhat of a surprise how useful the briquette-averaged data actually are. While a fine grid is needed in order to advance the solution in time with high accuracy, this same fine grid is not needed to represent the solution. Volume renderings from full-resolution images of the mixing fraction and the vorticity are compared with renderings based on the briquette-averaged lower-resolution data sets in Figure 8. The full-resolution data consist of a single-byte voxel value at each cell of the uniform Cartesian grid to represent each variable we wish to volume render. For the mixing fraction full resolution means double resolution, i.e., in this case 3072³ voxels, due to the subgrid resolving power of the higher-order PPB advection scheme (see the appendix of Woodward et al. 2015). The high-resolution rendering of the vorticity and other quantities is based on the grid-resolution data. The lower-resolution renderings of both mixing fraction and vorticity are based on data cubes with four times fewer grid points along each axis. The 384³ data cubes contain averages of $4\times 4\times 4$-cell briquettes, which for the mixing fraction represent $8\times 8\times 8$ significant data values. The images shown in the top row of Figure 8 use the full-resolution data and of course give the best representation. The most important role of renderings and 3D visualizations is to allow a qualitative assessment of the flow, which will ultimately guide quantitative model building. Even renderings based on the 64 times less voluminous briquette-averaged data shown in the bottom row still expose most of the key features of the flow.

Zoom In Zoom Out Reset image size

Figure 8 shows the far hemisphere of the He-shell flash convection or pulse-driven convective zone (similar to those shown in Figures 5 and 6; see Section 4.1) in a low-metallicity AGB star. The inner core, including the He shell and a stable layer above that contains H-rich, unprocessed envelope material, is included in the simulation volume. The gas is confined by gravity and also by a reflecting boundary sphere at a radius of 33.5 Mm. The star is shown in the midst of a global oscillation of shell hydrogen ingestion (GOSH; Herwig et al. 2014). H-rich gas is entrained into the pulse-driven convective zone from just above the top of the convection zone, at a radius of about 28 Mm. Waves of combustion involving this mixed-in H and the ${}_{}^{12}{\rm{C}}$ are propagating around the outer portion of the pulse-driven convective zone. The convection zone has been formed by the helium shell flash, with helium burning located at the bottom of the convection zone, around a radius of 13 Mm. The combustion of entrained gas at radii around 17 Mm drives strong local updrafts, which greatly enhance convective boundary mixing as the combustion waves propagate. This is of course best seen in a movie animation.

In Figure 8, two counterpropagating wave fronts have recently collided in the region of the lower left, and a clearly visible puff of entrained gas has been forced downward there, helping to form what will soon become a shell of H-enriched gas floating near the top of the convection zone. In the images at the left the gas of the star's carbon–oxygen core and the helium-and-carbon mixture of the convection zone are rendered as transparent. The color map represents only the H-rich gas component, which initially is confined to the stable layers above the convective boundary. At the point of the simulation shown, some amount of the H-rich fluid has been entrained into the pulse-driven convective zone. From the assumed viewing perspective one sees the lowest H-concentrations first. Where these, rendered dark blue, are sufficiently low, one sees through them into the more highly enriched gas. At this stage, the lower surface of the newly formed shell of H-enriched gas is fairly easily deformed as rising plumes of hotter, more buoyant gas tend to force it aside as they decelerate and ultimately reverse their upward flow. Ridges of dark blue in these images show the lanes that separate neighboring upwellings. These ridges of H-enriched gas are descending, pulling the gas from above the convection zone deeper into the hot region below, where the hydrogen will burn and drive new waves of combustion. All of these key features are clearly discernible in the low-resolution renderings shown in the bottom row of Figure 8.

Volume renderings of the vorticity magnitude (right column of Figure 8) reveal a thin, highly turbulent region at the front of the descending puff of entrained gas at the lower left. A typical, unstable shear layer induced by boundary-layer separation (Woodward et al. 2015) can be seen in the upper right quadrant. These features have been identified as an important component of the 3D entrainment mechanism at boundaries that are Kelvin–Helmholtz stable according to the 1D radial stratification. The images based on the briquette-averaged data give a fuzzy, slightly out-of-focus impression. But they still clearly reveal these key features of the flow and, when such images are animated, their dynamics.

Another more quantitative example of how the briquette-averaged data can be used is shown in Figure 9. Even at a moderately sized grid of 768³ the down-sampled data provide a good initial impression of the overall structure of the flow at a computational- and data-related cost that can be easily accommodated in the analysis scenario of the PPMstarhub. For the overall speed profile the low- and high-resolution data representations can hardly be discerned. Even when zooming in to the upper convective boundary, the low-resolution data provide meaningful exploratory information.

Zoom In Zoom Out Reset image size

We have used the cyberhubs technology to build the PPMstarhub application, which is addressing several issues. The PPMstar simulations of stellar convection are performed on the Blue Waters computing system at the NCSA center on as many as 400,000 cores. The resulting data sets are in their own way unique, similar to astronomical surveys. Although our team is exploiting them for their primary scientific purpose, there are potentially a number of additional questions that these data sets could answer. Sharing the raw data by just making them available for download is impractical owing to the size of the data, as well as the specialized analytic tools that are needed to access and explore the data. cyberhubs allows us to expose to interested users these simulation data sets together with our specialized software stack for analysis.

Over the years, a full range of tools have been developed at the LCSE that exploit the briquette data sets, both for visualization and for further analysis that would be involved, for example, in model building. As mentioned above, such models could be turbulence models, or mixing models, to ultimately be deployed in 1D stellar evolution models. These tools are now available along with access to several of our published data sets to interested users through PPMstarHub.³²

In addition, we have developed additional new Python-based analytic tools that work with the briquette data, as well as with single and multiple radial profile data. These, along with collections of example notebooks, are available on the PPMstar GitHub repository.³³ Specifically, the examples include notebooks that contain the analysis of all plots shown in our recent study on stellar hydrodynamics of O-shell convection in massive stars (Jones et al. 2017), as well as the notebooks of our project of simulations of low-Z AGB H-ingestion into a thermal-pulse He-shell flash (P. R. Woodward et al. 2018, in preparation). All of these example notebooks are staged on our PPMstarHub server, where the necessary data sets are staged as well, so that interested users can follow our data analysis. This is an example of how cyberhubs can play an essential role in making scientific analysis of raw simulation or observational data more transparent and accessible and the process of data analysis reproducible.