HACC Cosmological Simulations: First Data Release

We begin by describing the two large ΛCDM simulations run with identical cosmological parameters. The chosen parameter values are close to the best-fit WMAP-7 (Komatsu et al. 2011) cosmology, given by ω_cdm = 0.1109, ω_b = 0.02258, n_s = 0.963, h = 0.71, σ₈ = 0.8, and w₀ = −1.0. In the following, we provide details about the size of the simulations, the associated mass and force resolution, and the released data products.

2.1.1. The Outer Rim Simulation

The Outer Rim simulation, run some years ago, is still one of the largest simulations currently available at its mass resolution. It was carried out on Mira, an IBM BG/Q system at the ALCF. The simulation covers a volume of (4225 Mpc)³ and evolved 10240³ particles, leading to a mass resolution of ∼2.6 · 10⁹ M_⊙. (Note that while we quote the volume here in units of Mpc and M⊙, the data are delivered in units of h⁻¹ Mpc and h⁻¹ M_⊙.) The force resolution is approximately 4 kpc. (The Euclid Flagship simulation is similar, with a slightly lower mass resolution but increased volume.) The Outer Rim simulation was designed to be most useful for answering survey related questions and serve as the basis for generating detailed synthetic sky catalogs. The large volume enables the construction of large-area synthetic sky maps; at the same time the simulation has a good enough resolution to capture halos that host galaxies targeted by the surveys, such as luminous red galaxies and emission line galaxies.

Outer Rim simulation results have been used for a number of projects: halo catalogs for quasar clustering studies with eBOSS (Gil-Marín 2018; Hou et al. 2018; Zarrouk et al. 2018), a detailed investigation of the concentration–mass relation (Child et al. 2018), and strong lensing investigations (Li et al. 2019). The Outer Rim run is currently being used to create a major synthetic sky catalog for the second data challenge (DC2) carried out by the Large Synoptic Survey Telescope Dark Energy Science Collaboration (Korytov et al. 2019). Scientific results, some implementation details of HACC on Mira, the IBM BG/Q architecture, and a list of the stored outputs are given in Heitmann et al. (2019).

In future gravity-only simulations, we aim to improve the mass resolution by an order of magnitude (similar to the Q Continuum mass resolution) by increasing the number of particles while keeping the volume the same or increasing it slightly.

For the current data release, described here, we provide the following outputs at discrete time snapshots:

• 1.  
  Friends-of-friends halo properties (linking length b = 0.168)—the number of particles in the halo, a halo tag, the halo mass in units of h⁻¹ M_⊙, the potential minimum halo center (x, y, z) in h⁻¹ Mpc, the center of mass (x, y, z) in comoving h⁻¹ Mpc, the mean velocity (v_x, v_y, v_z) in peculiar comoving km s⁻¹, and the halo velocity dispersion. Note that the halo tag is not consistent across time steps.¹¹ The smallest halo contains 20 particles and therefore has a mass of ∼5.2 · 10¹⁰ M_⊙.
• 2.  
  1% of all particles in a simulation snapshot, randomly selected (positions in comoving h⁻¹ Mpc and velocities in peculiar comoving km s⁻¹).

The snapshots are stored at nine redshifts: z = {1.494, 1.433, 0.865, 0.779, 0.539, 0.502, 0.212, 0.050, 0.000}. The halos were identified with a standard Friends-of-Friends (FOF) halo finder (Davis et al. 1985) with a linking length of b = 0.168, a commonly used value for generation of synthetic galaxy catalogs.

In addition, we provide lightcone outputs in one octant of the sky for

• 1.  
  Full particle information (positions, velocities, and scale factor) out to redshift z ∼ 2.3.
• 2.  
  FOF halo properties (positions, velocities, scale factor, and mass) out to redshift $z\sim 3$.

All units are the same as for the snapshot data. The halo lightcones are created from 57 discrete time snapshots, while the particle lightcones are based on 49 snapshots. For the particle lightcone, particles with the same ID are identified in two adjacent snaphots and then the position of the particle where it intersects the lightcone is determined by linear interpolation. The volume of the Outer Rim simulation is not quite large enough to cover the full lightcone out to z = 3. We therefore employ standard replication procedures to enable the construction of a full octant out to z = 3. (For a detailed description of our approach, see Hollowed 2019; Korytov et al. 2019 and references therein.) For the halo lightcone, we use detailed merger trees to identify a halo's progenitor and then carry out an interpolation. A very detailed discussion of the procedure can be found in Hollowed (2019) and Korytov et al. (2019). As one example data product generated from the halo lightcone data, Figure 1 shows a small part of the SPHEREx sky (Doré et al. 2014), which is populated with galaxies using a halo occupation distribution method that takes into account different types of galaxies as seen by SPHEREx, a proposed all-sky near-infrared spectral survey. A fly-through movie is available on YouTube.¹²

Zoom In Zoom Out Reset image size

2.1.2. The Q Continuum Simulation

The Q Continuum simulation was carried out on Titan at the Oak Ridge Leadership Computing Facility, taking advantage of Titan's GPU/CPU architecture. Selected science results and implementation details for HACC on CPU/GPU platforms are presented in Heitmann et al. (2015). The simulation covers a smaller volume compared to the Outer Rim simulation ((1300 Mpc)³) but provides better mass resolution by more than an order of magnitude at ∼1.48 · 10⁸ M_⊙ (the force resolution is 3 kpc). The simulation evolved 8192³ particles and was designed to enable detailed structure formation studies at high mass resolution and very good statistics. In addition, while the volume is not large enough to enable the construction of large-scale sky survey maps, it offers a very valuable test bed for new modeling approaches to investigate the galaxy-halo connection. In the past, the Bolshoi simulation (Klypin et al. 2011) has been used very successfully for this purpose but due to the smaller box size of (250 h⁻¹ Mpc)³, has limits with regard to available halo statistics at higher masses. In Child et al. (2018) we present results for the concentration–mass relation from the Q Continuum simulation, taking advantage of the large number of halos available in different mass ranges at high mass resolution.

Figure 2 shows a zoomed-in view of the particle distribution from one MPI rank (the simulation was carried out using a total of 16,384 ranks) from the Q Continuum simulation at the final redshift z = 0. As in the case of the Outer Rim simulation, we release FOF halos at nine redshifts (the same redshifts were chosen for both simulations) and downsampled particle snapshots (1% randomly sampled as for the Outer Rim simulation). The smallest halos contain 40 particles and therefore have a mass of ∼5.92 · 10⁹ M_⊙.

Zoom In Zoom Out Reset image size

The Mira-Titan Universe suite—named after the two supercomputers it was generated on—is a unique set of simulations that spans a wide range of cosmological models. In the first public release we include data from 11 models that vary 7 parameters, namely $\theta =\{{\omega }_{m},{\omega }_{b},h,{\sigma }_{8},{n}_{s},{w}_{0},{w}_{a}\}$. All of these models and their parameters are listed in Table 1. Each simulation covers a (2100 Mpc)³ volume and evolves 3200³ particles. Figure 3 shows the results for the nonlinear density fluctuation power spectra at z = 0 for all 11 models.

Zoom In Zoom Out Reset image size

The complete Mira-Titan Universe suite also includes models with massive neutrinos, and results from these will be released in the near future. The suite encompasses 111 models and has been designed with a tessellation-based sampling strategy to build a range of cosmological emulators with known convergence properties. The model selection of the subset presented here is based on a symmetric Latin-Hypercube (SLH) design (for details on SLH designs in the context of cosmological emulators see Heitmann et al. 2009) for M001–M010 covering the following parameter ranges:

$$\begin{eqnarray}&&0.12\leqslant {\omega }_{m}\leqslant 0.155,\end{eqnarray} \tag{ 1 }$$

$$\begin{eqnarray}&&0.0215\leqslant {\omega }_{b}\leqslant 0.0235,\end{eqnarray} \tag{ 2 }$$

$$\begin{eqnarray}&&0.7\leqslant {\sigma }_{8}\leqslant 0.9,\end{eqnarray} \tag{ 3 }$$

$$\begin{eqnarray}&&0.55\leqslant h\leqslant 0.85,\end{eqnarray} \tag{ 4 }$$

$$\begin{eqnarray}&&0.85\leqslant {n}_{s}\leqslant 1.05,\end{eqnarray} \tag{ 5 }$$

$$\begin{eqnarray}&&-1.3\leqslant {w}_{0}\leqslant -0.7,\end{eqnarray} \tag{ 6 }$$

$$\begin{eqnarray}&&-1.73\leqslant {w}_{a}\leqslant 1.28.\end{eqnarray} \tag{ 7 }$$

The choices for these ranges are discussed in detail in Heitmann et al. (2016). For the parameterization of the dark energy equation of state, we follow the common definition introduced by Chevalier & Polarski (2001), Linder (2003): $w(a)={w}_{0}+{w}_{a}(1-a)$, where a = 1/(1 + z) is the expansion factor. We also provide results for a ΛCDM model, M000, using the same cosmological parameters as for the Outer Rim and Q Continuum simulations. Model M000 has been used in the past for a few science investigations already, including the generation of a galaxy power spectrum emulator (Kwan et al. 2015), simulations of the pairwise kinematic Sunyaev–Zel'dovich signal (Flender et al. 2016), and in the first Mira-Titan Universe paper to study requirements on the simulation volume and resolution to enable precision emulation of the matter power spectrum (Heitmann et al. 2016).

Since the data sets from the Mira-Titan Universe are smaller compared to the Outer Rim and Q Continuum simulations we release a broader set of outputs. We provide, for each of 27 redshifts, z = {4.000, 3.046, 2.478, 2.018, 1.799, 1.610, 1.376, 1.209, 1.006, 0.779, 0.736, 0.695, 0.656, 0.618, 0.578, 0.539, 0.502, 0.471, 0.434, 0.402, 0.364, 0.304, 0.242, 0.212, 0.154, 0.101, 0.000}, the following data products:

• 1.  
  Friends-of-friends halo properties: the same halo properties as for the Outer Rim simulation are provided, down to 20 particles per halo; halo masses depend on the exact cosmology but start at ∼10¹¹ M_⊙.
• 2.  
  Downsampled halo particles: for each halo we provide the positions and velocities for 1% of the halo particles, randomly chosen. If the halo size is smaller than 500 particles, we provide 5 particles instead of 1%.
• 3.  
  Full halo particles: for halos that have at least 1000 particles, we provide positions and velocities of all halo particles. This information overlaps with the downsampled particles but both catalogs can be used for different purposes and we therefore wanted to provide them both as a complete, independent set.
• 4.  
  1% of all particles in a simulation snapshot, randomly selected (positions in comoving h⁻¹ Mpc and velocities in peculiar comoving km s⁻¹).

The last column in Table 1 indicates the machine the simulation was run on (M: Mira, T: Titan) and the random seed for setting the initial condition. If the seed and the supercomputer are the same (as they are for M007–M010) the simulations have the same phases in the initial condition and can be compared directly, structure by structure, instead of just statistically via, e.g., the power spectra or mass functions. This feature is of potential interest for some comparative investigations.

Petrel is a flexible data service designed to support collaborative access to large data sets within scientific communities. Building on a high-performance 1.7 PB parallel file system (to be increased in the near future) and embedded in Argonne National Laboratory's 100+ Gbps network fabric, Petrel leverages Science DMZ concepts and Globus APIs to provide high-speed, highly connected, and programmatically controllable data management.

Petrel provides a user-oriented, collaborative storage model in which users manage their own isolated storage allocations. They are able to upload data to their allocation, manage the organization of data, and also dynamically share data without requiring local user accounts. It provides both Web and API access, via Globus, which allow sophisticated applications, such as the data portal described here, to be implemented with modest amounts of programming.

Petrel overcomes two important challenges associated with managing and using large data: high-speed data access and flexible collaborative usage. These aims have long been at odds with the way data is made accessible at high-performance computing (HPC) centers. For example, data are typically stored either on parallel file systems designed for rapid internal access or on specialized data portal servers that support slower external data distribution. Thus high-speed data movement in and out of HPC centers is often difficult. Second, dynamic collaboration is challenging due to the need for individual user accounts and policies regarding who may create accounts. In many cases the process for obtaining accounts lacks the flexibility required to support dynamic collaborations.

Petrel is designed to support implementations of the modern research data portal (MRDP) design pattern (Chard et al. 2018), in which control channel communications and data channel communications are separated, with the former handled by a web server computer deployed (most often) in the institution's enterprise network and the latter by specialized data servers connected directly to high-speed networks and storage systems. This pattern has emerged due to the availability of two technologies. The first, the Science DMZ (Dart et al. 2013), enables high-speed data access by connecting specialized data servers directly to high-speed networks and storage systems. The second, Globus, enables the outsourcing of crucial data portal functionality such as managing data transfers, data access, and authentication.

3.1.1. Petrel Data Store and DTNs

3.1.2. Globus Remote Access Services

The HACC Simulation Data Portal (shown in Figures 4 and 5) offers users an easy means of selecting simulation data and beginning a transfer to their home institutions for further analysis. This functionality is essential given that users are typically interested in analyzing particular subsets of the data. The portal permits selection of subsets of the data by simulation suite, model, redshift, and data type. In the case of the Mira-Titan Universe simulations, data is available from models simulated with 11 different sets of cosmological parameters, as enumerated in Table 1. Users can select a range of associated redshift values, and the data types of interest, and proceed to Globus, where they can select a destination for the transfer.

Zoom In Zoom Out Reset image size

Users can browse data on the portal without authenticating. Transfers are managed by Globus: after selecting data in the HACC Simulation Data Portal, the user is redirected to Globus to select a destination endpoint: typically, the user's home institution, but the user could, of course, instead transfer the data to any compute resource for analysis. Once the transfer has been submitted, the user is redirected back to a transfer status page on the HACC Simulation Data Portal server. Transfer activity can also be viewed on Globus.

In the example given in Figure 5, the user has chosen to download halo properties, full sets of halo particles, and downsampled simulation particles from eleven models, with one redshift value, resulting in a transfer of 2129 files, ranging in size from 4 KB to 794 MB (total 151 GB); this is a manageable subset of the available data. As a point of reference, transfers of this data set to the ALCF and the National Energy Research Scientific Computing Center (NERSC) completed in 48.2 s and 40.1 s, respectively. Smaller transfers are, of course, possible (for example, by selecting halo properties from a single model and a high redshift, when fewer halos would have formed). The portal maintains a record of downloads to aid our understanding of how the data are being used, so we can tailor current and future data releases.

Zoom In Zoom Out Reset image size

The portal is an implementation of the MRDP design pattern, which combines a modern web front end for browsing and discovering data, with Globus services on the server for managing data access and movement. This component-based approach makes it possible for the front-end portal to reside on publicly accessible server nodes to accept user requests, while calls to Globus enable direct data transfers from the Petrel DTNs, sitting in the Science DMZ, to the user's destination resource. We extended the example MRDP by developing mechanisms for creating and updating the catalogs of simulation data, developing functionality to enable querying and filtering the data, and creating an intuitive web interface to make these capabilities available to users. Our portal supports queries based on model and cosmological parameters, and selecting slices of the data across model and redshift values. The example MRDP was a very useful starting point for building a powerful data transfer portal, particularly in that the support for authentication and transfer worked without requiring any additional effort.

The setup of our HACC Simulation Data Portal allows the user to build a HACC simulation directory at their home institution over time. At the first requested data transfer, a directory structure is built with the following layout in the user chosen root directory, which could be called HACC_Simulations. At the top layer, the simulation suite or name will appear (for this release, this will be either OuterRim, QContinuum, or MiraTitanU). Next, a directory will be created with the model name as it appears on the web portal (e.g., M000), then a directory will appear that indicates the box size in Mpc. After that, a directory name that indicates the realization of the simulation will be built, e.g., HACC000. While in this release we have only one realization per model, in later releases this might change. In addition, this directory name helps to easily identify models that have been run with the same seed. For example, in Table 1, we indicate in the last column that the seeds for models M007–M010 in the Mira-Universe suite are the same. This is reflected in the HACC007 directory name. Finally, a data analysis directory, analysis is built that contains sub-directories for the halo, particle, or lightcone outputs (for the simulations they are available). Within those directories, more sub-directories are built depending on the user's data choices.

When the user requests the next transfer and points Globus Transfer to the same root directory, above HACC_Simulations, the overall directory structure will be preserved and the new data will be added accordingly. As a concrete example, if the user downloaded in the first transfer M000 data from MiraTitanU and now decides to investigate M002, the M002 directory will automatically be placed into the MiraTitanU directory. Another example could be that the user only requested halo properties during the first transfer for M000 and now also requests downsampled particles. In this case, the files will be automatically placed into the appropriate analysis directory for M000. This setup will help the user to keep a clean directory structure that can be built up over time automatically.

Data obtained from the HACC Simulation Data Portal uses the GenericIO data format. Efficient I/O is extremely important in the context of simulations at scale. In order to achieve optimal write speed, we developed a customized I/O library for HACC, called GenericIO. At simulation time, the GenericIO library, described in some detail in Habib et al. (2016), conducts a hierarchical aggregation of the outputs from the simulation ranks to match the design of the interconnect and I/O subsystem on a particular supercomputer. Instead of writing a single monolithic file of the particle data, which would be very inefficient, GenericIO subdivides the output into several files (depending on the simulation size, ${ \mathcal O }$(100)) in order to achieve a high percentage of the peak performance of the available bandwidth. The parameters that GenericIO uses for aggregation are tunable to accommodate the design of diverse I/O systems and different output sizes. For example, on Mira, a BG/Q architecture with a GPFS file system, GenericIO writes one file per dedicated I/O node in the system. On that system there are 128 compute nodes for each I/O node, leading to a 128-fold reduction in the number of output files.

The GenericIO library is open source and includes C++ and Python interfaces for accessing the data sets obtained from the portal. In order to build the library (see footnote 12) a reasonably modern C/C++ compiler is required; for example, the library builds successfully with GCC 4.8.5. Two build options are provided with GenericIO: serial tools that can run on the front end of a machine, including a Python library; and an MPI library that can be linked into a parallel executable.

The front-end programs can be built with the command:

$$\begin{eqnarray*}&&{\mathtt{make}}\ {\mathtt{frontend}} \mbox{-} {\mathtt{progs}}\end{eqnarray*}$$

This build includes two command-line executables. The GenericIOPrint executable is run as GenericIOPrint gio_datafile to view the file content as text; GenericIOVerify, executed similarly, is used to confirm the internal consistency of GenericIO-formatted data files. An example of building the frontend-progs target and a subsequent Python session is given in the Appendix.

The MPI programs can be built with the command

$$\begin{eqnarray*}&&{\mathtt{make}}\ {\mathtt{mpi}} \mbox{-} {\mathtt{progs}}\end{eqnarray*}$$

This build includes GenericIOPrint and GenericIOVerify, described above, and three other programs: GenericIORewrite, GenericIOBenchmarkRead, and GenericIOBenchmarkWrite.

The GenericIORewrite program is used to rewrite existing files with a different number of MPI ranks or with a subset of the original fields:

$$\begin{eqnarray*}&&\begin{array}{l}{\mathtt{GenericIORewrite}}\ {\mathtt{gio}}\_{\mathtt{old}}\ {\mathtt{gio}}\_{\mathtt{new}}\setminus \\ \qquad [{\mathtt{field}}{\mathtt{1}}\ [{\mathtt{field}}{\mathtt{2}}...]]\end{array}\end{eqnarray*}$$

The benchmark codes are used for benchmarking writes and reads to parallel file systems. The source code of the command-line tools serves as a good reference for how one might use the GenericIO library in an application; the benchmark codes are perhaps the most direct and compact of these examples.

As mentioned above, for each output GenericIO generates a set of files instead of one monolithic file. Each output, which could be a snapshot or a shell from the lightcones, will have a metadata file and a set of subfiles that are marked with a hash. When using the reader, one can either point it at the main, unhashed file and automatically the full set of files will be read, or one can read each hash file separately. In case of the snapshot files, the hash files contain blocks of data and each block originates from a rank and holds a contiguous region in space. However, the blocks within a hashed file are usually not from the same spatial region so that a single file will contain disconnected blocks that originate from different parts of the simulation volume. If the user works with one simulation but different data products, the regions within a hash file are the same across the different outputs. This is sometimes convenient if, e.g., one inspects a large halo that was found from the FOF properties files and wants to match the halo particles in the corresponding file.