Table of contents

It takes a village to perform a petascale computation—domain scientists, applied mathematicians, computer scientists, computer system vendors, program managers, and support staff—and the village was assembled during 24–28 June 2007 in Boston's Westin Copley Place for the third annual Scientific Discovery through Advanced Computing (SciDAC) 2007 Conference. Over 300 registered participants networked around 76 posters, focused on achievements and challenges in 36 plenary talks, and brainstormed in two panels. In addition, with an eye to spreading the vision for simulation at the petascale and to growing the workforce, 115 participants—mostly doctoral students and post-docs complementary to the conferees—were gathered on 29 June 2007 in classrooms of the Massachusetts Institute of Technology for a full day of tutorials on the use of SciDAC software. Eleven SciDAC-sponsored research groups presented their software at an introductory level, in both lecture and hands-on formats that included live runs on a local BlueGene/L.

Computation has always been about garnering insight into the behavior of systems too complex to explore satisfactorily by theoretical means alone. Today, however, computation is about much more: scientists and decision makers expect quantitatively reliable predictions from simulations ranging in scale from that of the Earth's climate, down to quarks, and out to colliding black holes. Predictive simulation lies at the heart of policy choices in energy and environment affecting billions of lives and expenditures of trillions of dollars. It is also at the heart of scientific debates on the nature of matter and the origin of the universe. The petascale is barely adequate for such demands and we are barely established at the levels of resolution and throughput that this new scale of computation affords. However, no scientific agenda worldwide is pushing the petascale frontier on all its fronts as vigorously as SciDAC.

The breadth of this conference archive reflects the philosophy of the SciDAC program, which was introduced as a collaboration of all of the program offices in the Office of Science of the U.S. Department of Energy (DOE) in Fall 2001 and was renewed for a second period of five years in Fall 2006, with additional support in certain areas from the DOE's National Nuclear Security Administration (NNSA) and the U.S. National Science Foundation (NSF). All of the projects in the SciDAC portfolio were represented at the conference and most are captured in this volume. In addition, the Organizing Committee incorporated into the technical program a number of computational science highlights from outside of SciDAC, and, indeed, from outside of the United States.

As implied by the title, scientific discovery is the driving deliverable of the SciDAC program, spanning the full range of the DOE Office of Science: accelerator design, astrophysics, chemistry and materials science, climate science, combustion, life science, nuclear physics, plasma physics, and subsurface physics. As articulated in the eponymous report that launched SciDAC, the computational challenges of these diverse areas are remarkably common. Each is profoundly multiscale in space and time and therefore continues to benefit at any margin from access to the largest and fastest computers available. Optimality of representation and execution requires adaptive, scalable mathematical algorithms in both continuous (geometrically complex domain) and discrete (mesh and graph) aspects. Programmability and performance optimality require software environments that both manage the intricate details of the underlying hardware and abstract them for scientific users. Running effectively on remote specialized hardware requires transparent workflow systems. Comprehending the petascale data sets generated in such simulations requires automated tools for data exploration and visualization. Archiving and sharing access to this data within the inevitably distributed community of leading scientists requires networked collaborative environments. Each of these elements is a research and development project in its own right. SciDAC does not replace theoretical programs oriented towards long-term basic research, but harvests them for contemporary, complementary state-of-the-art computational campaigns. By clustering researchers from applications and enabling technologies into coordinated, mission-driven projects, SciDAC accomplishes two ends with remarkable effectiveness: (1) it enriches the scientific perspective of both applications and enabling communities through mutual interaction and (2) it leverages between applications solutions and effort encapsulated in software.

Though SciDAC is unique, its objective of multiscale science at extreme computational scale is shared and approached through different programmatic mechanisms, notably NNSA's ASC program, NSF's Cyberinfrastructure program, and DoD's CREATE program in the U.S., and RIKEN's computational simulation programs in Japan. Representatives of each of these programs were given the podium at SciDAC 2007 and communication occurred that will be valuable towards the ends of complementarity, leverage, and promulgation of best practices. The 2007 conference was graced with additional welcome program announcements. Michael Strayer announced a new program of postdoctoral research fellowships in the enabling technologies. (The computer science post-docs will be named after the late Professor Ken Kennedy, who briefly led the SciDAC project Center for Scalable Application Development Software (CScADS) until his untimely death in February 2007.) IBM announced its petascale BlueGene/P system on June 26. Meanwhile, at ISC07 in Dresden, the semi-annual posting of a revised Top 500 list on June 27 showed several new Top 10 systems accessible to various SciDAC participants.

While SciDAC is dominated in 2007 by the classical scientific pursuit of understanding through reduction to components and isolation of causes and effects, simulation at scale is beginning to offer something even more tantalizing: synthesis and integration of multiple interacting phenomena in complex systems. Indeed, the design-oriented elements of SciDAC, such as accelerator and tokamak modeling, area already emphasizing multiphysics coupling, and climate science has been doing so for years in the coupling of models of the ocean, atmosphere, ice, and land. In one of the panels at SciDAC 2007, leaders of a three-stage `progressive workshop' on exascale simulation for energy and environment (E3), considered prospects for whole-system modeling in a variety of scientific areas within the domain of DOE related to energy, environmental, and global security. Computer vendors were invited to comment on the prospects for delivering exascale computing systems in another panel. The daunting nature of this challenge is summarized with the observation that the peak processing power of the entire Top 500 list of June 2007 is only 0.0052 exaflop/s. It takes the combined power of most of the computers on the internet today worldwide to reach 1 exaflop/s or 10¹⁸ floating point operations per second.

The program of SciDAC 2007 followed a template honed by its predecessor meetings in San Francisco in 2005 and Denver in 2006. The Boston venue permitted outreach to a number of universities in the immediate region and throughout southern New England, including SciDAC campuses of Boston University, Harvard, and MIT, and a dozen others including most of the Ivy League. Altogether 55 universities, 20 laboratories, 14 private companies, 5 agencies, and 4 countries were represented among the conference and tutorial workshop participants. Approximately 47% of the conference participants were from government laboratories, 37% from universities, 9% from federal program offices, and 7% from industry.

Keys to the success of SciDAC 2007 were the informal poster receptions, coffee breaks, working breakfasts and lunches, and even the `Right-brain Night' featuring artistic statements, both reverent and irreverent, by computational scientists, inspired by their work. The organizers thank the sponsors for their generosity in attracting participants to these informal occasions with sumptuous snacks and beverages: AMD, Cray, DataDirect, IBM, SGI, SiCortex, and the Institute of Physics.

A conference as logistically complex as SciDAC 2007 cannot possibly and should not be executed primarily by the scientists, themselves. It is a great pleasure to acknowledge the many talented staff that contributed to a productive time for all participants and nearperfect adherence to schedule. Chief among them is Betsy Riley, currently detailed from ORNL to the program office in Germantown, with degrees in mathematics and computer science, but a passion for organizing interdisciplinary scientific programs. Betsy staffed the organizing committee during the year of telecon meetings leading up to the conference and masterminded sponsorship, invitations, and the compilation of the proceedings. Assisting her from ORNL in managing the program were Daniel Pack, Angela Beach, and Angela Fincher. Cynthia Latham of ORNL performed admirably in website and graphic design for all aspects of the online and printed materials of the meeting. John Bui, John Smith, and Missy Smith of ORNL ran their customary tight ship with respect to audio-visual execution and capture, assisted by Eric Ecklund and Keith Quinn of the Westin. Pamelia Nixon-Hartje of Ambassador Services was personally invaluable in getting the most out of the hotel and its staff. We thank Jeff Nichols of ORNL for managing the primary subcontract for the meeting.

The SciDAC tutorial program was a joint effort of Professor John Negele of MIT, David Skinner, PI of the SciDAC Outreach Center, and the SciDAC 2007 Chair. Sponsorship from the Outreach Center in the form of travel scholarships for students, and of the local area SciDAC university delegation of BU, Harvard, and MIT for food and facilities is gratefully acknowledged.

Of course, the archival success of a scientific meeting rests with the willingness of the presenters to make the extra effort to package their field-leading science in a form suitable for interaction with colleagues from other disciplines rather than fellow specialists. This goal, oft-stated in the run up to the meeting, was achieved to an admirable degree, both in the live presentations and in these proceedings. This effort is its own reward, since it leads to enhanced communication and accelerated scientific progress.

Our greatest thanks are reserved for Michael Strayer, Associate Director for OASCR and the Director of SciDAC, for envisioning this celebratory meeting three years ago, and sustaining it with his own enthusiasm, in order to provide a highly visible manifestation of the fruits of SciDAC. He and the other Office of Science program managers in attendance and working in Washington, DC to communicate the opportunities afforded by SciDAC deserve the gratitude of a new virtual scientific village created and cemented under the vision of scientific discovery through advanced computing.

David E Keyes Fu Foundation Professor of Applied Mathematics

Good morning. Welcome to Boston, the home of the Red Sox, Celtics and Bruins, baked beans, tea parties, Robert Parker, and SciDAC 2007.

A year ago I stood before you to share the legacy of the first SciDAC program and identify the challenges that we must address on the road to petascale computing—a road E E Cummins described as `. . . never traveled, gladly beyond any experience.' Today, I want to explore the preparations for the rapidly approaching extreme scale (X-scale) generation. These preparations are the first step propelling us along the road of burgeoning scientific discovery enabled by the application of X- scale computing. We look to petascale computing and beyond to open up a world of discovery that cuts across scientific fields and leads us to a greater understanding of not only our world, but our universe.

As part of the President's America Competitiveness Initiative, the ASCR Office has been preparing a ten year vision for computing. As part of this planning the LBNL together with ORNL and ANL hosted three town hall meetings on Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security (E3). The proposed E3 initiative is organized around four programmatic themes:

• Engaging our top scientists, engineers, computer scientists and applied mathematicians;

• investing in pioneering large-scale science;

• developing scalable analysis algorithms, and storage architectures to accelerate discovery; and

• accelerating the build-out and future development of the DOE open computing facilities.

It is clear that we have only just started down the path to extreme scale computing. Plan to attend Thursday's session on the out-briefing and discussion of these meetings.

The road to the petascale has been at best rocky. In FY07, the continuing resolution provided 12% less money for Advanced Scientific Computing than either the President, the Senate, or the House. As a consequence, many of you had to absorb a no cost extension for your SciDAC work. I am pleased that the President's FY08 budget restores the funding for SciDAC. Quoting from Advanced Scientific Computing Research description in the House Energy and Water Development Appropriations Bill for FY08,

"Perhaps no other area of research at the Department is so critical to sustaining U.S. leadership in science and technology, revolutionizing the way science is done and improving research productivity."

As a society we need to revolutionize our approaches to energy, environmental and global security challenges. As we go forward along the road to the X-scale generation, the use of computation will continue to be a critical tool along with theory and experiment in understanding the behavior of the fundamental components of nature as well as for fundamental discovery and exploration of the behavior of complex systems. The foundation to overcome these societal challenges will build from the experiences and knowledge gained as you, members of our SciDAC research teams, work together to attack problems at the tera- and peta- scale. If SciDAC is viewed as an experiment for revolutionizing scientific methodology, then a strategic goal of ASCR program must be to broaden the intellectual base prepared to address the challenges of the new X-scale generation of computing. We must focus our computational science experiences gained over the past five years on the opportunities introduced with extreme scale computing.

Our facilities are on a path to provide the resources needed to undertake the first part of our journey. Using the newly upgraded 119 teraflop Cray XT system at the Leadership Computing Facility, SciDAC research teams have

• in three days performed a 100-year study of the time evolution of the atmospheric CO2 concentration originating from the land surface. The simulation of the El Nino/Southern Oscillation which was part of this study has been characterized as `the most impressive new result in ten years'

• gained new insight into the behavior of superheated ionic gas in the ITER reactor as a result of an AORSA run on 22,500 processors that achieved over 87 trillion calculations per second (87 teraflops) which is 74% of the system's theoretical peak.

Tomorrow, Argonne and IBM will announce that the first IBM Blue Gene/P, a 100 teraflop system, will be shipped to the Argonne Leadership Computing Facility later this fiscal year. By the end of FY2007 ASCR high performance and leadership computing resources will include the 114 teraflop IBM Blue Gene/P; a 102 teraflop Cray XT4 at NERSC and a 119 teraflop Cray XT system at Oak Ridge.

Before ringing in the New Year, Oak Ridge will upgrade to 250 teraflops with the replacement of the dual core processors with quad core processors and Argonne will upgrade to between 250–500 teraflops, and next year, a petascale Cray Baker system is scheduled for delivery at Oak Ridge.

The multidisciplinary teams in our SciDAC Centers for Enabling Technologies and our SciDAC Institutes must continue to work with our Scientific Application teams to overcome the barriers that prevent effective use of these new systems. These challenges include:

• the need for new algorithms as well as operating system and runtime software and tools which scale to parallel systems composed of hundreds of thousands processors;

• program development environments and tools which scale effectively and provide ease of use for developers and scientific end users; and

• visualization and data management systems that support moving, storing, analyzing, manipulating and visualizing multi-petabytes of scientific data and objects.

The SciDAC Centers, located primarily at our DOE national laboratories will take the lead in ensuring that critical computer science and applied mathematics issues are addressed in a timely and comprehensive fashion and to address issues associated with research software lifecycle.

In contrast, the SciDAC Institutes, which are university-led centers of excellence, will have more flexibility to pursue new research topics through a range of research collaborations. The Institutes will also work to broaden the intellectual and researcher base—conducting short courses and summer schools to take advantage of new high performance computing capabilities.

The SciDAC Outreach Center at Lawrence Berkeley National Laboratory complements the outreach efforts of the SciDAC Institutes. The Outreach Center is our clearinghouse for SciDAC activities and resources and will communicate with the high performance computing community in part to understand their needs for workshops, summer schools and institutes.

SciDAC is not ASCR's only effort to broaden the computational science community needed to meet the challenges of the new X-scale generation. I hope that you were able to attend the Computational Science Graduate Fellowship poster session last night. ASCR developed the fellowship in 1991 to meet the nation's growing need for scientists and technology professionals with advanced computer skills. CSGF, now jointly funded between ASCR and NNSA, is more than a traditional academic fellowship. It has provided more than 200 of the best and brightest graduate students with guidance, support and community in preparing them as computational scientists. Today CSGF alumni are bringing their diverse top-level skills and knowledge to research teams at DOE laboratories and in industries such as Proctor and Gamble, Lockheed Martin and Intel. At universities they are working to train the next generation of computational scientists.

To build on this success, we intend to develop a wholly new Early Career Principal Investigator's (ECPI) program. Our objective is to stimulate academic research in scientific areas within ASCR's purview especially among faculty in early stages of their academic careers. Last February, we lost Ken Kennedy, one of the leading lights of our community. As we move forward into the extreme computing generation, his vision and insight will be greatly missed. In memorial to Ken Kennedy, we shall designate the ECPI grants to beginning faculty in Computer Science as the Ken Kennedy Fellowship. Watch the ASCR website for more information about ECPI and other early career programs in the computational sciences.

We look to you, our scientists, researchers, and visionaries to take X-scale computing and use it to explode scientific discovery in your fields. We at SciDAC will work to ensure that this tool is the sharpest and most precise and efficient instrument to carve away the unknown and reveal the most exciting secrets and stimulating scientific discoveries of our time. The partnership between research and computing is the marriage that will spur greater discovery, and as Spencer said to Susan in Robert Parker's novel, `Sudden Mischief', `We stick together long enough, and we may get as smart as hell'.

Michael Strayer

It is often said that numerical simulation is third in the group of three ways to explore modern science: theory, experiment and simulation. Carefully executed modern numerical simulations can, however, be considered at least as relevant as experiment and theory. In comparison to physical experimentation, with numerical simulation one has the numerically simulated values of every field variable at every grid point in space and time. In comparison to theory, with numerical simulation one can explore sets of very complex non-linear equations such as the Einstein equations that are very difficult to investigate theoretically. Cyber-enabled scientific discovery is not just about numerical simulation but about every possible issue related to scientific discovery by utilizing cyberinfrastructure such as the analysis and storage of large data sets, the creation of tools that can be used by broad classes of researchers and, above all, the education and training of a cyber-literate workforce.

Particle in cell (PIC) methods are effective in computing Vlasov-Poisson system of equations used in simulations of magnetic fusion plasmas. PIC methods use grid based computations, for solving Poisson's equation or more generally Maxwell's equations, as well as Monte-Carlo type methods to sample the Vlasov equation. The presence of two types of discretizations, deterministic field solves and Monte-Carlo methods for the Vlasov equation, pose challenges in understanding and optimizing performance on today large scale computers which require high levels of concurrency. These challenges arises from the need to optimize two very different types of processes and the interactions between them. Modern cache based high-end computers have very deep memory hierarchies and high degrees of concurrency which must be utilized effectively to achieve good performance. The effective use of these machines requires maximizing concurrency by eliminating serial or redundant work and minimizing global communication. A related issue is minimizing the memory traffic between levels of the memory hierarchy because performance is often limited by the bandwidths and latencies of the memory system. This paper discusses some of the performance issues, particularly in regard to parallelism, of PIC methods. The gyrokinetic toroidal code (GTC) is used for these studies and a new radial grid decomposition is presented and evaluated. Scaling of the code is demonstrated on ITER sized plasmas with up to 16K Cray XT3/4 cores.

Supporting the visualization and analysis needs of the users of the Department of Energy's premiere high-performance computing centers requires a careful engineering of software and hardware system architectures to provide maximum capability and algorithmic breadth. Data set growth follows an inverse power law that has implications for the platforms that are deployed for analysis and visualization; central storage and coupled analysis platforms are critical for petascale post-production. Software architectures like VisIt – which exploit parallel platforms, as well as provide remote capability, extensibility, and optimization – are fruitful ground for delivering new analysis capabilities for petascale applications. Finally, direct interaction with customers is key to deploying successful results.

The broad scientific objectives of the SWIM (Simulation of Wave Interaction with MHD) project are: (A) To improve our understanding of interactions that both RF wave and particle sources have on extended-MHD phenomena, and to substantially improve our capability for predicting and optimizing the performance of burning plasmas in devices such as ITER: and (B) To develop an integrated computational system for treating multi-physics phenomena with the required flexibility and extensibility to serve as a prototype for the Fusion Simulation Project (FSP).

Using examples from active research areas in combustion and astrophysics, we demonstrate a computationally efficient numerical approach for simulating multiscale low Mach number reacting flows. The method enables simulations that incorporate an unprecedented range of temporal and spatial scales, while at the same time, allows an extremely high degree of reaction fidelity. Sample applications demonstrate the efficiency of the approach with respect to a traditional time-explicit integration method, and the utility of the methodology for studying the interaction of turbulence with terrestrial and astrophysical flame structures.

This talk describes a new project in SciDAC II in the area of low-energy nuclear physics. The motivation and goals of the SciDAC are presented as well as an outline of the theoretical and computational methodology that will be employed. An important motivation is to have more accurate and reliable predictions of nuclear properties including their binding energies and low-energy reaction rates. The theoretical basis is provided by density functional theory, which the only available theory that can be systematically applied to all nuclei. However, other methodologies based on wave function methods are needed to refine the functionals and to make applications to dynamic processes.

Progress is reviewed on the simulation of wave-particle interactions in the ion cyclotron range of frequencies (ICRF). Two important aspects of this problem are described. First, mode conversion from a long wavelength fast magnetosonic wave to short wavelength ion Bernstein waves (IBW) and ion cyclotron waves (ICW) is simulated and validation tests of the simulations against experiment are presented. Second, simulations of the quasilinear evolution of nonthermal ion tails during the minority heating are reviewed and experimental validation tests are also discussed. In this paper we describe how access to teraflop computing capability has made it possible to advance the state of the art in this area. We also discuss two aspects of the wave-particle interaction where future work is needed and where in particular access to sub-petaflop and petaflop computing capability would be highly desirable. This work involves the interaction of ICRF waves with energetic neutral beam ions at high ion cyclotron harmonic number and addresses the inclusion of finite ion drift orbit effects in the nonthermal ion tail evolution and the inclusion of nonlinear effects such as RF sheaths in the antenna – edge plasma coupling.

Scientific datasets obtained by measurement or produced by computational simulations must be analyzed to understand the phenomenon under study. The analysis typically requires a mathematically sound definition of the features of interest and robust algorithms to identify these features, compute statistics about them, and often track them over time. Because scientific datasets often capture phenomena with multi-scale behaviour, and almost always contain noise the definitions and algorithms must be designed with sufficient flexibility and care to allow multi-scale analysis and noise-removal. In this paper, we present some recent work on topological feature extraction and tracking with applications in molecular analysis, combustion simulation, and structural analysis of porous materials.

This work reports on studies of the multi-scale interaction between small-scale electron turbulence and large-scale ion turbulence in tokamak plasmas. Traditionally, the long-wavelength, low-frequency turbulence driven by ion-scale instabilities (ion-temperature-gradient and trapped-electron modes) is studied separately from the short-wavelength, high-frequency turbulence driven by electron-scale instabilities (electron-temperature-gradient modes). High-resolution, massively-parallel simulations have uncovered a number of physically-important discoveries. First, we find that a popular simplified model of ion physics previously used in studies of electron-scale turbulence can lead to nonphysical runaway of electron heat transport. We have shown that this nonphysical runaway is eliminated when correct long-wavelength ion physics is self-consistently included. We have also found that under normal conditions most of the electron heat transport arises from large-scale instabilities. However, when these large-scale instabilities are suppressed by plasma rotation or other processes, the electron instabilities survive and may dominate the loss of electron heat from the plasma. A further remarkable finding is that strong turbulence at long scales can act to reduce the intensity of turbulence at short scales. Simulations were carried out on the Cray X1E computer at ORNL, with the largest runs taking about a week on 720 multi-streaming processors.

Accelerators are the largest and most costly scientific instruments of the Department of Energy, with uses across a broad range of science, including colliders for particle physics and nuclear science and light sources and neutron sources for materials studies. COMPASS, the Community Petascale Project for Accelerator Science and Simulation, is a broad, four-office (HEP, NP, BES, ASCR) effort to develop computational tools for the prediction and performance enhancement of accelerators. The tools being developed can be used to predict the dynamics of beams in the presence of optical elements and space charge forces, the calculation of electromagnetic modes and wake fields of cavities, the cooling induced by comoving beams, and the acceleration of beams by intense fields in plasmas generated by beams or lasers. In SciDAC-1, the computational tools had multiple successes in predicting the dynamics of beams and beam generation. In SciDAC-2 these tools will be petascale enabled to allow the inclusion of an unprecedented level of physics for detailed prediction.

The final merger of comparable mass binary black holes produces an intense burst of gravitational radiation and is one of the strongest sources for both ground-based and space-based gravitational wave detectors. Since the merger occurs in the strong-field dynamical regime of general relativity, numerical relativity simulations of the full Einstein equations in 3-D are required to calculate the resulting gravitational dynamics and waveforms. While this problem has been pursued for more than 30 years, the numerical codes have long been plagued by various instabilities and, overall, progress was incremental. Recently, however, dramatic breakthrough have occurred, resulting in robust simulations of merging black holes. In this paper, we examine these developments and the exciting new results that are emerging.

The Genomics Revolution has resulted in a massive and growing quantity of whole-genome DNA sequences, which encode the metabolic catalysts necessary for life. However, gene annotations can rarely be complete, and measurement of the kinetic constants associated with the encoded enzymes can not possibly keep pace, necessitating the use of careful modeling to explore plausible network behaviors. Key challenges are (1) quantitatively formulating kinetic laws governing each transformation in a fixed model network; (2) characterizing the stable solution (if any) of the associated ordinary differential equations (ODEs); (3) fitting the latter to metabolomics data as it becomes available; and, (4) optimizing a model output against the possible space of kinetic parameters, with respect to properties such as robustness of network response, or maximum consumption/production. This SciDAC-2 project addresses this large-scale uncertainty in the genome-scale metabolic network of the water-splitting, H₂-producing green alga Chlamydomonas reinhardtii. Each metabolic transformation is formulated as an irreversible steady-state process, such that the vast literature on known enzyme mechanisms may be incorporated directly. To start, glycolysis, the tricarboxylic acid cycle, and basic fermentation pathways have been encoded in Systems Biology Markup Language (SBML) with careful annotation and consistency with the KEGG database, yielding a model with 3 compartments, 95 species, 38 reactions, and 109 kinetic constants. To study and optimize such models with a view toward larger models, we have developed a system which takes as input an SBML model, and automatically produces C code that when compiled and executed optimizes the model's kinetic parameters according to test criteria. We describe the system and present numerical results. Further development, including overlaying of a parallel multistart algorithm, will allow optimization of thousands of parameters on high-performance systems ranging from distributed grids to unified petascale architectures.

With petascale applications comes petascale data; gleaning knowledge and insight from this large-scale data is widely accepted to be a limiting factor in many fields of scientific endeavor. Large-scale data presents two incredible challenges: scale and complexity. The scale challenge refers to the problem of being able to process tremendous numbers of bytes of data within some time constraint. The complexity challenge is more subtle; petascale data is inherently complex and reducing this data to comprehensible forms is difficult.

For the scale challenge, it is often assumed that continued usage of techniques viable at the terascale will provide a recipe for success at the petascale. We argue the opposite: following this path will result in cost prohibitive solutions. Instead, we must pursue 'smarter' techniques, such as in situ and multi-resolution processing, which are well established by a decade's worth of research.

The forms of postprocessing are very diverse, but they can be summarized in broad use cases: data exploration, presentation graphics, quantitative analysis, comparative analysis, and visual debugging. Together, these use cases are responsive to the complexity challenge. Moreover, these use cases are responsive to the demands of the user community. Making pretty pictures is just one component of the simulation community's postprocessing needs and all of these needs must be addressed for petascale data.

A major focus of this paper is to construct a road map for realizing this broad set of use cases at the petascale, and, further, to do this in the most economical way possible. None of the smart processing techniques we consider are a panacea; none are capable of supporting every use case. However, a software architecture that underpins these processing techniques and that can dynamically select between them will provide an economical way to meet these challenging requirements, and in this paper we discuss such an architecture.

In this paper, we discuss some of the issues in obtaining high performance for block-structured adaptive mesh refinement software for partial differential equations. We show examples in which AMR scales to thousands of processors. We also discuss a number of metrics for performance and scalability that can provide a basis for understanding the advantages and disadvantages of this approach.

The paper provides an overview of the challenges involved in the computation of flows with interactions between turbulence, strong shockwaves, and sharp density interfaces. The prediction and physics of such flows is the focus of an ongoing project in the Scientific Discovery through Advanced Computing (SciDAC) program. While the project is fundamental in nature, there are many important potential applications of scientific and engineering interest ranging from inertial confinement fusion to exploding supernovae. The essential challenges will be discussed, and some representative numerical results that highlight these challenges will be shown. In addition, the overall approach taken in this project will be outlined.

SciDAC applications have a demonstrated need for advanced software tools to manage the complexities associated with sophisticated geometry, mesh, and field manipulation tasks, particularly as computer architectures move toward the petascale. The Center for Interoperable Technologies for Advanced Petascale Simulations (ITAPS) will deliver interoperable and interchangeable mesh, geometry, and field manipulation services that are of direct use to SciDAC applications. The premise of our technology development goal is to provide such services as libraries that can be used with minimal intrusion into application codes. To develop these technologies, we focus on defining a common data model and data-structure neutral interfaces that unify a number of different services such as mesh generation and improvement, front tracking, adaptive mesh refinement, shape optimization, and solution transfer operations. We highlight the use of several ITAPS services in SciDAC applications.

We present magnetohydrodynamic (MHD) models of stellar core collapse computed with our astrophysical MHD code GenASiS. Microphysics in the models include a realistic equation of state and a simple parametrization of deleptionization during collapse. We give a brief description of some numerical results, and discuss requirements for capturing the physics of the magnetorotational instability in the context of core collapse supernovae.

In this paper we study the use of long distance interpolation methods with the low complexity coarsening algorithm PMIS. AMG performance and scalability is compared for classical as well as long distance interpolation methods on parallel computers. It is shown that the increased interpolation accuracy largely restores the scalability of AMG convergence factors for PMIS-coarsened grids, and in combination with complexity reducing methods, such as interpolation truncation, one obtains a class of parallel AMG methods that enjoy excellent scalability properties on large parallel computers.

We describe some recent mathematical results in constructing computational methods that lead to the development of fast and accurate multiresolution numerical methods for solving quantum chemistry and nuclear physics problems based on Density Functional Theory (DFT). Using low separation rank representations of functions and operators in conjunction with representations in multiwavelet bases, we developed a multiscale solution method for integral and differential equations and integral transforms. The Poisson equation, the Schrodinger equation, and the projector on the divergence free functions provide important examples with a wide range of applications in computational chemistry, nuclear physics, computational electromagnetic and fluid dynamics.

We have implemented this approach along with adaptive representations of operators and functions in the multiwavelet basis and low separation rank (LSR) approximation of operators and functions. These methods have been realized and implemented in a software package called Multiresolution Adaptive Numerical Evaluation for Scientific Simulation (MADNESS).

Early scattering experiments revealed that the proton was not a point particle but a bound state of many quarks and gluons. Deep inelastic scattering (DIS) experiments have accurately determined the probability of struck quarks carrying a fraction of the proton's momentum. The current generation of experiments and Lattice QCD calculations will provide detailed multi-dimensional pictures of the distributions of quarks and gluons inside the proton.

Petascale science is an end-to-end endeavour, involving not only the creation of massive datasets at supercomputers or experimental facilities, but the subsequent analysis of that data by a user community that may be distributed across many laboratories and universities. The new SciDAC Center for Enabling Distributed Petascale Science (CEDPS) is developing tools to support this end-to-end process. These tools include data placement services for the reliable, high-performance, secure, and policy-driven placement of data within a distributed science environment; tools and techniques for the construction, operation, and provisioning of scalable science services; and tools for the detection and diagnosis of failures in end-to-end data placement and distributed application hosting configurations. In each area, we build on a strong base of existing technology and have made useful progress in the first year of the project. For example, we have recently achieved order-of-magnitude improvements in transfer times (for lots of small files) and implemented asynchronous data staging capabilities; demonstrated dynamic deployment of complex application stacks for the STAR experiment; and designed and deployed end-to-end troubleshooting services. We look forward to working with SciDAC application and technology projects to realize the promise of petascale science.

Laser driven wakefield accelerators produce accelerating fields thousands of times those achievable in conventional radio-frequency accelerators, offering compactness and ultrafast bunches to potentially extend the frontiers of high energy physics and enable laboratory scale ultrafast radiation sources. Realization of this potential requires understanding of accelerator physics to advance beam performance and stability, and particle simulations model the highly nonlinear, kinetic physics required. One-to-one simulations of experiments provide new insight for optimization and development of 100 MeV to GeV and beyond laser accelerator stages, and on production of reproducible and controllable low energy spread beams with improved emittance (focusability) and energy through control of injection.

With petascale computers only a year or two away there is a pressing need to anticipate and compensate for a probable increase in failure and application interruption rates. Researchers, designers and integrators have available to them far too little detailed information on the failures and interruptions that even smaller terascale computers experience. The information that is available suggests that application interruptions will become far more common in the coming decade, and the largest applications may surrender large fractions of the computer's resources to taking checkpoints and restarting from a checkpoint after an interruption. This paper reviews sources of failure information for compute clusters and storage systems, projects failure rates and the corresponding decrease in application effectiveness, and discusses coping strategies such as application-level checkpoint compression and system level process-pairs fault-tolerance for supercomputing. The need for a public repository for detailed failure and interruption records is particularly concerning, as projections from one architectural family of machines to another are widely disputed. To this end, this paper introduces the Computer Failure Data Repository and issues a call for failure history data to publish in it.

The quark-gluon plasma is a novel state of matter in which quarks are no longer confined to bound states such as baryons and mesons. Just after the Big Bang the Universe was filled with a quark-gluon plasma. The cores of neutron stars may well be a quark-gluon plasma. At the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory, scientists collide high energy gold beams to produce and study the quark gluon plasma. Lattice gauge theorists use computers to study to properties of the quark-gluon plasma such as the temperature required to produce it and its equation of state. These demanding calculations will require over a Petaflop-year of computing to achieve theoretical goals.

The enormous and growing complexity of today's high-end systems has increased the already significant challenge of maximizing performance on today's equally complex scientific applications. In this paper, we discuss the role of compiler technology in supporting application developers in a systematic approach to performance tuning of key application components. Based on scenarios taken from manual optimization of scientific codes, we describe how compiler support can enable the programmer to achieve the same or better performance in a much more productive way. We also present examples derived automatically from compiler optimization that show results comparable to hand-tuned performance.

Numerical modeling is a critical tool to the U.S. Department of Energy for evaluating the environmental impact of remediation strategies for subsurface legacy waste sites. Unfortunately, the physical and chemical complexity of many sites overwhelms the capabilities of even most state of the art groundwater models. Of particular concern is the representation of highly-heterogeneous stratified rock/soil layers in the subsurface and the biological and geochemical interactions of chemical species within multiple fluid phases. There is clearly a need for higher-resolution modeling (i.e. increased spatial and temporal resolution) and increasingly mechanistic descriptions of subsurface physicochemical processes (i.e. increased chemical degrees of freedom).

We present SciDAC-funded research being performed in furthering the development of PFLOTRAN, a parallel multiphase flow and multicomponent reactive transport model. Written in Fortran90, PFLOTRAN is founded upon PETSc data structures and solvers. We are employing PFLOTRAN to simulate uranium transport at the Hanford 300 Area, a contaminated site of major concern to the Department of Energy, the State of Washington, and other government agencies. By leveraging the billions of degrees of freedom available through high-performance computation using tens of thousands of processors, we can better characterize the release of uranium into groundwater and its subsequent transport to the Columbia River, and thereby better understand and evaluate the effectiveness of various proposed remediation strategies.

This paper describes the Carbon-Land Model Intercomparison Project (C-LAMP) being carried out through a collaboration between the Community Climate System Model (CCSM) Biogeochemistry Working Group, a DOE SciDAC-2 project, and the DOE Program for Climate Model Diagnosis and Intercomparison (PCMDI). The goal of the project is to intercompare terrestrial biogeochemistry models running within the CCSM framework to determine the best set of processes to include in future versions of CCSM. As a part of the project, observational datasets are being collected and used to score the scientific performance of these models following a well-defined set of metrics. In addition, metadata standards for terrestrial biosphere models are being developed to support archival and distribution of the C-LAMP model output via the Earth System Grid (ESG). Progress toward completion of this project and preliminary results from the first set of experiments are reported.

The Performance Engineering Research Institute (PERI) project focuses on achieving superior performance for Scientific Discovery through Advanced Computing (SciDAC) applications on leadership class machines through cutting-edge research in performance modeling and automated performance tuning. This focus requires coordinated activities to engage SciDAC application teams. The initial application engagement activity was a survey of these teams to determine their performance goals, the criticality of those goals, current performance of their applications, application characteristics relevant to performance and their plans for future optimization. Using a web-based questionnaire, PERI researchers have worked with application developers to provide this information for over twenty-five applications. This paper describes the initial analysis of the application characteristics and performance goals, as well as current and future application engagement activities driven by these results. While the survey was conducted primarily to meet PERI needs, the results represent a snapshot of the state of SciDAC code development and may be of use to the DOE community at large. Overall, the results show that SciDAC application teams are engaged in significant new code development, which will require flexible performance optimization techniques that can improve performance as the applications evolve.

Physical constraints such as power, leakage and pin bandwidth are currently driving the HPC industry to produce systems with unprecedented levels of concurrency. In these parallel systems, synchronization and memory operations are becoming considerably more expensive than before. In this work we study parallel matrix factorization codes and conclude that they need to be re-engineered to avoid unnecessary (and expensive) synchronization. We propose the use of multithreading combined with intelligent schedulers and implement representative algorithms in this style. Our results indicate that this strategy can significantly outperform traditional codes.

This paper presents our recent progress in terascale three-dimensional simulations of turbulent nonpremixed flames in the presence of a mean flow strain and fine water droplets. Under the ongoing university collaborative project supported by the DOE SciDAC Program [1] along with the INCITE 2007 Project [2], the study aims at bringing the state-of-the-art high-fidelity simulation capability to the next level by incorporating various advanced physical models for soot formation, radiative heat transfer, and lagrangian spray dynamics, to an unprecedented degree of detail in high-fidelity simulation application. The targeted science issue is fundamental characteristics of flame suppression by the complex interaction between turbulence, chemistry, radiation, and water spray. The high quality simulation data with full consideration of multi-physics processes will allow fundamental understanding of the key physical and chemical mechanisms in the flame quenching behavior. In this paper, recent efforts on numerical algorithms and model development toward the targeted terascale 3D simulations are discussed and some preliminary results are presented.

Global climate models need to simulate several small, regional-scale processes which affect the global circulation in order to accurately simulate the climate. This is particularly important in the ocean where small scale features such as oceanic eddies are currently represented with adhoc parameterizations. There is also a need for higher resolution to provide climate predictions at small, regional scales. New high-performance computing platforms such as the IBM BlueGene can provide the necessary computational power to perform ultra-high resolution climate model integrations. We have begun to investigate the scaling of the individual components of the Community Climate System Model to prepare it for integrations on BlueGene and similar platforms. Our investigations show that it is possible to successfully utilize (32K) processors. We describe the scalability of five models: the Parallel Ocean Program (POP), the Community Ice CodE (CICE), the Community Land Model (CLM), and the new CCSM sequential coupler (CPL7) which are components of the next generation Community Climate System Model (CCSM); as well as the High-Order Method Modeling Environment (HOMME) which is a dynamical core currently being evaluated within the Community Atmospheric Model. For our studies we concentrate on 1/10° resolution for CICE, POP, and CLM models and 1/4° resolution for HOMME. The ability to simulate high resolutions on the massively parallel petascale systems that will dominate high-performance computing for the foreseeable future is essential to the advancement of climate science.

Many technologically important materials display complex behavior due to strong electronic correlations. These include magnets, magnetoresistive materials, lanthanides and actinides and high temperature superconductors. Some of these materials display complex phase diagrams with multiple competing ground states, spin and charge ordering and complex excitations like those associated with spin and charge separation. An unbiased treatment of these phenomena requires a theory able to treat the correlations over all relevant length scales. In part, due to the minus sign problem, numerically exact methods employed to study these systems scale exponentially. Thus, despite great recent progress such as the development of the dynamical mean field approximation and its cluster extensions, there is as yet no unbiased study of these complex phenomena. The Multi-Scale Many-Body (MSMB) formalism and algorithms address these problems by separating the correlations into different length scales and treating each with an appropriate approximation: strong correlations at short length scales are treated by numerically exact Quantum Monte Carlo (QMC) simulations, those at intermediate length scales are treated by Feynman diagrammatic methods, while the weak correlations at long length scales are treated by a (dynamical) mean field approximation with a self-consistently determined effective medium.

The Visualization and Analytics Center for Enabling Technologies (VACET) focuses on leveraging scientific visualization and analytics software technology as an enabling technology for increasing scientific productivity and insight. Advances in computational technology have resulted in an 'information big bang,' which in turn has created a significant data understanding challenge. This challenge is widely acknowledged to be one of the primary bottlenecks in contemporary science. The vision of VACET is to adapt, extend, create when necessary, and deploy visual data analysis solutions that are responsive to the needs of DOE's computational and experimental scientists. Our center is engineered to be directly responsive to those needs and to deliver solutions for use in DOE's large open computing facilities. The research and development directly target data understanding problems provided by our scientific application stakeholders. VACET draws from a diverse set of visualization technology ranging from production quality applications and application frameworks to state-of-the-art algorithms for visualization, analysis, analytics, data manipulation, and data management.

Gyrokinetic particle simulations are critical to the study of anomalous energy transport associated with plasma microturbulence in magnetic confinement fusion experiments. The simulations are conducted on massively parallel computers and produce large quantities of particles, variables, and time steps, thus presenting a formidable challenge to data analysis tasks. We present two new visualization techniques for scientists to improve their understanding of the time-varying, multivariate particle data. One technique allows scientists to examine correlations in multivariate particle data with tightly coupled views of the data in both physical space and variable space, and to visually identify and track features of interest. The second technique, built into SCIRun, allows scientists to perform range-based queries over a series of time slices and visualize the resulting particles using glyphs. The ability to navigate the multiple dimensions of the particle data, as well as query individual or a collection of particles, enables scientists to not only validate their simulations but also discover new phenomena in their data.

We present work carried out by the USQCD Collaboration on Software Infrastructure funded under SciDAC 2. We present successes of the software from the original SciDAC 1 project as well as ongoing and future work. We outline the various scientific collaborations SciDAC-2 has created.

The challenges of visualization at the extreme scale involve issues of scale, complexity, temporal exploration and uncertainty. The Visualization and Analytics Center for Enabling Technologies (VACET) focuses on leveraging scientific visualization and analytics software technology as an enabling technology to increased scientific discovery and insight. In this paper, we introduce new uses of visualization frameworks through the introduction of Equivalence Class Functions (ECFs). These functions give a new class of derived quantities designed to greatly expand the ability of the end user to explore and visualize data. ECFs are defined over equivalence classes (i.e., groupings) of elements from an original mesh, and produce summary values for the classes as output. ECFs can be used in the visualization process to directly analyze data, or can be used to synthesize new derived quantities on the original mesh. The design of ECFs enable a parallel implementation that allows the use of these techniques on massive data sets that require parallel processing.

The goal of this SciDAC project is to develop a scalable parallel and distributed computational framework consisting of methods, algorithms, and integrated software tools for: 1) multi Tera-to-Petascale simulations with quantum-level accuracy; 2) multimillion-to-multibillion-to-trillion atom molecular dynamics (MD) simulations based on density functional theory (DFT) and temperature dependent model generalized pseudopotential theory; 3) quasicontinuum (QC) method embedded with classical atomistic and quantum simulations based on DFT; and 4) accelerated molecular dynamics (AMD) coupled with hierarchical atomistic/QC simulations to reach macroscopic length and time scales relevant to SCC. Scalability is being achieved beyond 10⁵ processors through linear-scaling algorithms and performance-optimization techniques. We are employing automated model transitioning to embed higher fidelity simulations concurrently inside coarser simulations only when and where they are required. Each scale and model has well-defined error bounds with controlled error propagation across the scales and models to estimate uncertainty in predictions.

While the disruption of a star in a supernova explosion lasts only seconds, the ejected stellar material, much of it newly radioactive, continues to shine brightly for several months. Multi-physics simulations of supernova are now illuminating the physics of the stellar explosion itself, but further post-processing of models with a radiative transfer code is needed to predict the visible emission and compare directly to astronomical observations. Here we discuss Monte Carlo techniques for addressing multi-group time-dependent radiative transfer in 3-dimensional, rapidly expanding atmospheres. We have developed the SEDONA code, which calculates detailed angle dependent light curves, spectra, and polarization, and have applied it to study the nature and cosmological utility of supernovae. The code is scalable and is typically run on high-performance computers using 1024 processors.

Scientists often face the need for more computing power than is available locally, but are constrained by the fact that even if the required resources were available remotely, their complex software stack would not be easy to port to those resources. Many applications are dependency-rich and complex, making it hard to run them on anything but a dedicated platform. Worse, even if the applications do run on another system, the results they produce may not be consistent across different runs.

As part of the Center for Enabling Distributed Petascale Science (CEDPS) project we have been developing the workspace service which allows authorized clients to dynamically provision execution environments, using virtual machine technology, on remote computers. Virtual machines provide an excellent implementation of a portable environment as they allow users to configure an environment and then deploy it on a variety of platforms. This paper describes a proof-of-concept of this strategy developed for the High-Energy and Nuclear Physics (HENP) applications such as STAR's. We are currently building on this work to enable production STAR runs in virtual machines.

The accuracy and convergence of computational solutions of mesh-based methods is strongly dependent on the quality of the mesh used. We have developed methods for optimizing meshes that are comprised of elements of arbitrary polygonal and polyhedral type. We present in this research the development of r-h hybrid adaptive meshing technology tailored to application areas relevant to multi-physics modeling and simulation. Solution-based adaptation methods are used to reposition mesh nodes (r-adaptation) or to refine the mesh cells (h-adaptation) to minimize solution error. The numerical methods perform either the r-adaptive mesh optimization or the h-adaptive mesh refinement method on the initial isotropic or anisotropic meshes to equidistribute weighted geometric and/or solution error function. We have successfully introduced r-h adaptivity to a least-squares method with spherical harmonics basis functions for the solution of the spherical shallow atmosphere model used in climate modeling. In addition, application of this technology also covers a wide range of disciplines in computational sciences, most notably, time-dependent multi-physics, multi-scale modeling and simulation.

The SciDAC2 accelerator project at SLAC aims to simulate an entire three-cryomodule radio frequency (RF) unit of the International Linear Collider (ILC) main Linac. Petascale computing resources supported by advances in Applied Mathematics (AM) and Computer Science (CS) and INCITE Program are essential to enable such very large-scale electromagnetic accelerator simulations required by the ILC Global Design Effort. This poster presents the recent advances and achievements in the areas of CS/AM through collaborations.

TOPS is providing high-performance, scalable sparse direct solvers, which have had significant impacts on the SciDAC applications, including fusion simulation (CEMM), accelerator modeling (COMPASS), as well as many other mission-critical applications in DOE and elsewhere. Our recent developments have been focusing on new techniques to overcome scalability bottleneck of direct methods, in both time and memory. These include parallelizing symbolic analysis phase and developing linear-complexity sparse factorization methods. The new techniques will make sparse direct methods more widely usable in large 3D simulations on highly-parallel petascale computers.

CO₂ sequestration (capture, separation, and long term storage) in various geologic media including depleted oil reservoirs, saline aquifers, and oceanic sediments is being considered as a possible solution to reduce green house gas emissions. Dissolution of supercritical CO₂ in formation brines is considered an important storage mechanism to prevent possible leakage. Accurate prediction of the plume dissolution rate and migration is essential. Analytical analysis and numerical experiments have demonstrated that convective instability (Rayleigh instability) has a crucial effect on the dissolution behavior and subsequent mineralization reactions. Global stability analysis indicates that a certain grid resolution is needed to capture the features of density-driven fingering phenomena. For 3-D field scale simulations, high resolution leads to large numbers of grid nodes, unfeasible for a single workstation. In this study, we investigate the effects of convective instability on geologic sequestration of CO₂ by taking advantage of parallel computing using the code PFLOTRAN, a massively parallel 3-D reservoir simulator for modeling subsurface multiphase, multicomponent reactive flow and transport based on continuum scale mass and energy conservation equations. The onset, development and long-term fate of a supercritical CO₂ plume will be resolved with high resolution numerical simulations to investigate the rate of plume dissolution caused by fingering phenomena.

The growing power of parallel supercomputers gives scientists the ability to simulate more complex problems at higher fidelity, leading to many high-impact scientific advances. To maximize the utilization of the vast amount of data generated by these simulations, scientists also need scalable solutions for studying their data to different extents and at different abstraction levels. As we move into peta- and exa-scale computing, simply dumping as much raw simulation data as the storage capacity allows for post-processing analysis and visualization is no longer a viable approach. A common practice is to use a separate parallel computer to prepare data for subsequent analysis and visualization. A naive realization of this strategy not only limits the amount of data that can be saved, but also turns I/O into a performance bottleneck when using a large parallel system. We conjecture that the most plausible solution for the peta- and exa-scale data problem is to reduce or transform the data in-situ as it is being generated, so the amount of data that must be transferred over the network is kept to a minimum. In this paper, we discuss different approaches to in-situ processing and visualization as well as the results of our preliminary study using large-scale simulation codes on massively parallel supercomputers.

Over the last 15 years, lattice discretizations of quarks that preserve chiral symmetry have progressed from beautiful theoretical ideas to practical simulation tools. With control over chiral symmetry, lattice calculations of more of the matrix elements relevant to the violation of charge-conjugation and parity (CP) symmetry in the Standard Model can be done. The progression to practical simulation tool has been aided by the continued growth in computing power, as well as marked advances in algorithms. The status of simulations with chiral quarks will be reviewed, emphasizing the physics of CP violation that is currently being done. The prospects for future simulations with physical values for the light quarks will also be discussed.

Nonlinear transport algorithms designed to reduce numerical diffusion fail to preserve correlations between moments, isotope abundances, etc. when these scalar densities are transported in models as separate tracers. In case of the particle size/composition coordinates of an aerosol, such loss can give rise to unphysical moment sets. New statistical approaches to aerosol dynamics, which involve tracking moments directly, offer highly efficient alternatives to sectional and modal methods for representing aerosols in climate models, but it is essential that moment set integrity be preserved throughout a simulation. In this paper we review the problem and weaknesses of previous attempts at solution, including vector transport – a scheme in which the moments, as internal aerosol coordinates, are transported together with a single lead tracer such as number or mass. A non-negative least squares (NNLS) solution that finally eliminates the problem without requiring modification of the transport algorithm itself is presented. Following each transport step, new moment sets are resolved into sums of previously validated sets with non-negative coefficients using NNLS Transport errors are removed and the now guaranteed-to-be-valid moment sets are ready for passage to the aerosol dynamics module. In addition to moment set validation, the new scheme reduces numerical diffusion during transport and provides greater accuracy for the source apportionment of aerosol mixtures. The method is not limited to moment transport – similar improvements in accuracy are expected using NNLS in conjunction with modal and sectional methods.

This paper introduces three component technology initiatives within the SciDAC Center for Technology for Advanced Scientific Component Software (TASCS) that address ever-increasing productivity challenges in creating, managing, and applying simulation software to scientific discovery. By leveraging the Common Component Architecture (CCA), a new component standard for high-performance scientific computing, these initiatives tackle difficulties at different but related levels in the development of component-based scientific software: (1) deploying applications on massively parallel and heterogeneous architectures, (2) investigating new approaches to the runtime enforcement of behavioral semantics, and (3) developing tools to facilitate dynamic composition, substitution, and reconfiguration of component implementations and parameters, so that application scientists can explore tradeoffs among factors such as accuracy, reliability, and performance.

A number of particle models that are suitable for simulating multiphase fluid flow and biogeochemical processes have been developed during the last few decades. Here we discuss three of them: a microscopic model – molecular dynamics; a mesoscopic model – dissipative particle dynamics; and a macroscopic model – smoothed particle hydrodynamics. Particle methods are robust and versatile, and it is relatively easy to add additional physical, chemical and biological processes into particle codes. However, the computational efficiency of particle methods is low relative to continuum methods. Multiscale particle methods and hybrid (particle–particle and particle–continuum) methods are needed to improve computational efficiency and make effective use of emerging computational capabilities. These new methods are under development.

As part of the US Department of Energy's Scientific Discovery through Advanced Computing (SciDAC-2) program, science teams are tackling problems that require computational simulation and modeling at the petascale. A grand challenge for computer science is to develop software technology that makes it easier to harness the power of these systems to aid scientific discovery. As part of its activities, the SciDAC-2 Center for Scalable Application Development Software (CScADS) is building open source software tools to support efficient scientific computing on the emerging leadership-class platforms. In this paper, we describe two tools for performance analysis and tuning that are being developed as part of CScADS: a tool for analyzing scalability and performance, and a tool for optimizing loop nests for better node performance. We motivate these tools by showing how they apply to S3D, a turbulent combustion code under development at Sandia National Laboratory. For S3D, our node performance analysis tool helped uncover several performance bottlenecks. Using our loop nest optimization tool, we transformed S3D's most costly loop nest to reduce execution time by a factor of 2.94 for a processor working on a 50³ domain.

CHIMERA is a multi-dimensional radiation hydrodynamics code designed to study core-collapse supernovae. The code is made up of three essentially independent parts: a hydrodynamics module, a nuclear burning module, and a neutrino transport solver combined within an operator-split approach. We describe some major algorithmic facets of the code and briefly discuss some recent results. The multi-physics nature of the problem, and the specific implementation of that physics in CHIMERA, provide a rather straightforward path to effective use of multi-core platforms in the near future.

The recent release of the Intergovernmental Panel on Climate Change (IPCC) 4th Assessment Report (AR4) has generated significant media attention. Much has been said about the US role in this report, which included significant support from the Department of Energy through the Scientific Discovery through Advanced Computing (SciDAC) and other Department of Energy (DOE) programs for climate model development and the production execution of simulations. The SciDAC-supported Earth System Grid Center for Enabling Technologies (ESG-CET) also played a major role in the IPCC AR4: all of the simulation data that went into the report was made available to climate scientists worldwide exclusively via the ESG-CET At the same time as the IPCC AR4 database was being developed, the National Center for Atmospheric Research (NCAR), a leading US climate science laboratory and a ESG participant, began publishing model runs from the Community Climate System Model (CCSM), and its predecessor the Parallel Coupled Model (PCM) through ESG In aggregate, ESG-CET provides seamless access to over 180 terabytes of distributed climate simulation data to over 6,000 registered users worldwide, who have taken delivery of more than 250 terabytes from the archive. Not only does this represent a substantial advance in scientific knowledge, it is also a major step forward in how we conduct the research process on a global scale. Moving forward, the next IPCC assessment report, AR5, will demand multi-site metadata federation for data discovery and cross-domain identity management for single sign-on of users in a more diverse federation enterprise environment. Towards this aim, ESG is leading the effort in the climate community towards standardization of material for the global federation of metadata, security, and data services required to standardize, analyze, and access data worldwide.

We describe PFLOTRAN, a recently developed code for modeling multi-phase, multi-component subsurface flow and reactive transport using massively parallel computers. PFLOTRAN is built on top of PETSc, the Portable, Extensible Toolkit for Scientific Computation. Leveraging PETSc has allowed us to develop—with a relatively modest investment in development effort—a code that exhibits excellent performance on the largest-scale supercomputers. Very significant enhancements to the code are planned during our SciDAC-2 project. Here we describe the current state of the code, present an example of its use on Jaguar, the Cray XT3/4 system at Oak Ridge National Laboratory consisting of 11706 dual-core Opteron processor nodes, and briefly outline our future plans for the code.

We present a brief overview of optimization tools that are being developed for SciDAC applications. We emphasize derivative-free and gradient-based methods since these tools make minimal demands on the user and the application. We discuss the performance of these tools and point out developments that have led to significant improvements in performance. A parameter estimation problem that arises in nuclear fission is used to illustrate the challenges that arise as we attack nonlinear, noisy, computationally-intensive optimization applications.

This paper discusses methods being developed by the ITAPS center to support the execution of parallel adaptive simulations on unstructured meshes. The paper first outlines the ITAPS approach to the development of interoperable mesh, geometry and field services to support the needs of SciDAC application in these areas. The paper then demonstrates the ability of unstructured adaptive meshing methods built on such interoperable services to effectively solve important physics problems. Attention is then focused on ITAPs' developing ability to solve adaptive unstructured mesh problems on massively parallel computers.

We present a new tabulation strategy for the numerical integration of chemical reacting flow processes on the basis of a non-stiff system of equations. Both the tabulation and the identification of the non-stiff system are adaptive and are based on the Computational Singular Perturbation (CSP) method. The tabulation strategy is implemented in order to store and reuse the CSP quantities required for the construction of the non-stiff model. In this paper we describe a particular feature of this algorithm, the 'homogeneous correction', that allows for an accurate and efficient identification of the manifold on which the solution moves according to the slow time scales. The improved efficiency in constructing the slow model and simulating the system dynamics along the manifold during run-time calculations is demonstrated.

At the Institute for Ultrascale Visualization, we are tackling the broad problem of building visualization solutions for the petascale age and beyond. As computing transitions into a new age where scalar solutions no longer improve in performance, and parallel solutions are the vehicle for future performance gains, one key challenge in our effort is to harness the power of emerging commodity data-parallel processors.

One of the most promising new parallel processors is the graphics processing unit (GPU), whose recent gains in programmability and enormous arithmetic and memory bandwidth substantially outpace its CPU counterpart. The GPU's performance and programmability, together with its graphics capabilities, make it particularly well suited for demanding next-generation visualization applications.

We face a number of important challenges in making the GPU a first-class citizen for computation in these visualization applications. In this paper, we describe how we are addressing these challenges: mapping GPUs to general-purpose tasks, fundamental algorithms for GPUs, visualization applications that use the GPU for general-purpose computation, and extending our applications to support multiple GPUs. We also describe trends in upcoming GPU designs and how they will impact our research moving forward.

Simulations in the subsurface environment represent a broad range of phenomena covering an equally broad range of scales. Developing modelling capabilities that can integrate models representing different phenomena acting at different scales present formidable challenges both from the algorithmic and computer science perspective. This paper will describe the development of an integrated framework that will be used to combine different models into a single simulation. Initial work has focused on creating two frameworks, one for performing smooth particle hydrodynamics (SPH) simulations of fluid systems, the other for performing grid-based continuum simulations of reactive subsurface flow. The SPH framework is based on a parallel code developed for doing pore scale simulations, the continuum grid-based framework is based on the STOMP (Subsurface Transport Over Multiple Phases) code developed at PNNL Future work will focus on combining the frameworks together to perform multiscale, multiphysics simulations of reactive subsurface flow.

The Open Science Grid (OSG) provides a distributed facility where the Consortium members provide guaranteed and opportunistic access to shared computing and storage resources. OSG provides support for and evolution of the infrastructure through activities that cover operations, security, software, troubleshooting, addition of new capabilities, and support for existing and engagement with new communities. The OSG SciDAC-2 project provides specific activities to manage and evolve the distributed infrastructure and support it's use. The innovative aspects of the project are the maintenance and performance of a collaborative (shared & common) petascale national facility over tens of autonomous computing sites, for many hundreds of users, transferring terabytes of data a day, executing tens of thousands of jobs a day, and providing robust and usable resources for scientific groups of all types and sizes. More information can be found at the OSG web site: www.opensciencegrid.org.

The Combinatorial Scientific Computing and Petascale Simulations (CSCAPES) Institute is developing algorithms and software for combinatorial problems that play an enabling role in scientific and engineering computations. Discrete algorithms will be increasingly critical for achieving high performance for irregular problems on petascale architectures. This paper describes recent contributions by researchers at the CSCAPES Institute in the areas of load balancing, parallel graph coloring, performance improvement, and parallel automatic differentiation.

Since last year, with funding from the Japanese government, RIKEN has been conducting a project to build a next generation supercomputer, which will be capable of carrying out peta or higher scale simulations and will be in full operation by the end of FY2011. Its main objectives are to keep leading edge hardware technology inside Japan, and also to stimulate the use of simulations in R&D in academia and industry in order to investigate sophisticated natural phenomena in ways which are only possible with such large scale simulations, for example, designing novel functional materials to create future industrial technologies. To promote this kind of use of the supercomputer, nano-science and life sciences have been selected as grand challenges, in which specially designed supercomputer codes are to be developed. This paper presents the key features of the project.

A code and methodology are introduced for solving the fully general relativistic magnetohydrodynamic (GRMHD) equations using time-explicit, finite-volume discretization. The code has options for solving the GRMHD equations using traditional artificial-viscosity (AV) or non-oscillatory central difference (NOCD) methods, or a new extended AV (eAV) scheme using artificial-viscosity together with a dual energy-flux-conserving formulation. The dual energy approach allows for accurate modeling of highly relativistic flows at boost factors well beyond what has been achieved to date by standard artificial viscosity methods. It provides the benefit of Godunov methods in capturing high Lorentz boosted flows but without complicated Riemann solvers, and the advantages of traditional artificial viscosity methods in their speed and flexibility. Additionally, the GRMHD equations are solved on an unstructured grid that supports local adaptive mesh refinement using a fully threaded oct-tree (in three dimensions) network to traverse the grid hierarchy across levels and immediate neighbors. Some recent studies will be summarized.

Modelling biological networks is inherently data-driven and data-intensive. The combinatorial nature of this type of modelling, however, requires new methods capable of dealing with the enormous size and irregularity of the search. Searching via 'backtracking' is one possible solution that avoids exhaustive searches by constraining the search space to the subspace of feasible solutions. Despite its wide use in many combinatorial optimization problems, there are currently few parallel implementations of backtracking capable of effectively dealing with the memory-intensive nature of the process and the extremely unbalanced loads present. In this paper, a parallel, scalable, and memory-efficient backtracking algorithm within the context of maximal clique enumeration is presented, and its applicability to large-scale biological networks aimed at studying the mechanisms for efficient bioethanol production is discussed.

We present the results of fully 3D adaptive mesh MHD simulations of fueling pellets injected into tokamaks. The Chombo framework for block structured local adaptive mesh refinement (AMR), extended to use the equilibrium magnetic coordinates, is employed to mitigate the problems due to the large range of spatial scales. Generalized upwinding techniques are employed to deal with sharp gradients. The modeling includes a semi-analytical kinetic treatment of the transport of electron energy flux which drives the ablation. We discuss the phenomenology of the mass redistribution processes involving the density equilibrating along field lines and transport across surfaces (in the large-major-radius direction).

Many subsurface flow and transport problems of importance today involve coupled non-linear flow, transport, and reaction in media exhibiting complex heterogeneity. In particular, problems involving biological mediation of reactions fall into this class of problems. Recent experimental research has revealed important details about the physical, chemical, and biological mechanisms involved in these processes at a variety of scales ranging from molecular to laboratory scales. However, it has not been practical or possible to translate detailed knowledge at small scales into reliable predictions of field-scale phenomena important for environmental management applications. A large assortment of numerical simulation tools have been developed, each with its own characteristic scale. Important examples include 1. molecular simulations (e.g., molecular dynamics); 2. simulation of microbial processes at the cell level (e.g., cellular automata or particle individual-based models); 3. pore-scale simulations (e.g., lattice-Boltzmann, pore network models, and discrete particle methods such as smoothed particle hydrodynamics); and 4. macroscopic continuum-scale simulations (e.g., traditional partial differential equations solved by finite difference or finite element methods). While many problems can be effectively addressed by one of these models at a single scale, some problems may require explicit integration of models across multiple scales. We are developing a hybrid multi-scale subsurface reactive transport modeling framework that integrates models with diverse representations of physics, chemistry and biology at different scales (sub-pore, pore and continuum). The modeling framework is being designed to take advantage of advanced computational technologies including parallel code components using the Common Component Architecture, parallel solvers, gridding, data and workflow management, and visualization. This paper describes the specific methods/codes being used at each scale, techniques used to directly and adaptively couple across model scales, and preliminary results of application to a multi-scale model of mineral precipitation at a solute mixing interface.

Applying subsurface simulation codes to understand heterogeneous flow and transport problems is a complex process potentially involving multiple models, multiple scales, and spanning multiple scientific disciplines. A typical end-to-end process involves many tools, scripts and data sources usually shared only though informal channels. Additionally, the process contains many sub-processes that are repeated frequently and could be automated and shared. Finally, keeping records of the models, processes, and correlation between inputs and outputs is currently manual, time consuming and error prone. We are developing a software framework that integrates a workflow execution environment, shared data repository, and analysis and visualization tools to support development and use of new hybrid subsurface simulation codes. We are taking advantage of recent advances in scientific process automation using the Kepler system and advances in data services based on content management. Extensibility and flexibility are key underlying design considerations to support the constantly changing set of tools, scripts, and models available. We describe the architecture and components of this system with early examples of applying it to a continuum subsurface model.

The importance of water in many areas of science has motivated an enormous number of experimental and theoretical investigations. In this paper, we review several recent examples where first-principles molecular dynamics simulations coupled with large scale computing has enabled important insight into the unique properties of water in a variety of systems, including pure liquid water, the solvation of ions and hydrophobic solutes and nanoscale confined systems such as water inside carbon nanotubes.

The past few years has seen a sea change in computer architecture that will impact every facet of our society as every electronic device from cell phone to supercomputer will need to confront parallelism of unprecedented scale. Whereas the conventional multicore approach (2, 4, and even 8 cores) adopted by the computing industry will eventually hit a performance plateau, the highest performance per watt and per chip area is achieved using manycore technology (hundreds or even thousands of cores). However, fully unleashing the potential of the manycore approach to ensure future advances in sustained computational performance will require fundamental advances in computer architecture and programming models that are nothing short of reinventing computing. In this paper we examine the reasons behind the movement to exponentially increasing parallelism, and its ramifications for system design, applications and programming models.

The Born-Oppenheimer separation of the Schroedinger equation allows the electronic and nuclear motions to be solved in three steps. 1) The solution of the electronic wave function at a discrete set of molecular conformations; 2) the fitting of this discrete set of energy values in order to construct an analytical approximation to the potential energy surface (PES) at all molecular conformations; 3) the use of this analytical PES to solve for the nuclear motion using either time-dependent or time-independent formulations to compute molecular energy values, chemical reaction rates, and cumulative reaction probabilities. This project involves the development of technology to address all three of these steps. This report focuses on our recent work on the optimization of nonlinear wave function parameters for the electronic wave functions.

With the increasing volume and complexity of data produced by ultra-scale simulations and high-throughput experiments, understanding the science is largely hampered by the lack of comprehensive, end-to-end data management solutions ranging from initial data acquisition to final analysis and visualization. The SciDAC-1 Scientific Data Management (SDM) Center succeeded in bringing an initial set of advanced data management technologies to DOE application scientists in astrophysics, climate, fusion, and biology. Equally important, it established collaborations with these scientists to better understand their science as well as their forthcoming data management and data analytics challenges. Our future focus is on improving the SDM framework to address the needs of ultra-scale science during SciDAC-2. Specifically, we are enhancing and extending our existing tools to allow for more interactivity and fault tolerance when managing scientists' workflows, for better parallelism and feature extraction capabilities in their data analytics operations, and for greater efficiency and functionality in users' interactions with local parallel file systems, active storage, and access to remote storage. These improvements are necessary for the scalability and complexity challenges presented by hardware and applications at ultra scale, and are complemented by continued efforts to work with application scientists in various domains.

Results from tokamak experiments such as PPPL's NSTX indicate that significant anomalous power absorption can occur in the edge of the fusion plasma. Understanding of this phenomenon is a critical issue for analysis of RF heating scenarios on the ITER fusion experiment. Two probable edge absorption candidates, rf sheath losses and parametric decay instability, are both inherently non-linear, and likely to depend significantly on non-axi-symmetric geometric detail in the vicinity of the antenna structures. Analysis of these phenomenon is beyond the capabilities of existing axi-symmetric frequency-domain linear-solvers used for analysis of heating and current drive in core fusion plasma, and so we are augmenting our analysis capability with the time-domain 3-D general-geometry electromagnetic and particle-in-cell simulation framework, Vorpal [1]. This framework is a modern object-oriented software package, which has demonstrated fast scalable operation on clusters of over 1000 cpu's, a necessity for this type of calculation. We have successfully introduced into this framework an implicit plasma solver [2], in order to accurately treat electromagnetic plasma wave characteristics in the wide range of plasma conditions occurring from edge plasma to core plasma, including situations where the plasma frequency is not resolvable at the rf time-scales of interest, and including sharp plasma resonances and cutoff behaviours common in the rf regime. We present benchmarking of this new plasma solver for 1-D, 2-D, and 3-D scenarios. We also discuss implementation plans for non-linear sheath boundary models, non-linear edge-plasma conditions leading to parametric decay, and also tracking of high-energy particles in core-heating scenarios, where issues of finite-banana-width effects and superadiabaticity remain outside the scope of the existing frequency-domain solvers.

Large-scale numerical computation is applied to fluid-based plasma modeling of edge localized modes (ELMs). The modes are magnetohydrodynamic-like instabilities that nonlinearly release bursts of particles and energy across the flow-induced transport barrier that lies near the separatrix of closed and open magnetic flux in tokamaks. Some of the first nonlinear computations of ELMs to model the separate drifts of ions and electrons, which linearly stabilize the highest wavenumber modes and avoid a nonlinear 'ultraviolet catastrophe' without the use of ad hoc dissipation, are presented. The two-fluid system supports high-frequency dispersive modes that are not present in magnetohydrodynamics, and a new implicit leapfrog algorithm has been implemented to run these computations. We briefly describe the algorithm and computational requirements, in addition to physical results on nonlinear structure formation.

The dynamics of galactic nuclei containing multiple supermassive black holes is modelled including relativistic dynamics. It is shown that for certain initial conditions there is no stalling problem for the relativistic coalescence of supermassive black hole binaries. This astrophysical application and another one using a smoothed particle hydrodynamics code are our first use cases on a new computer architecture using GRAPE and new MPRACE accelerator cards based on reconfigurable chips, developed in the GRACE project. We briefly discuss our science applications and first benchmarks obtained with the new hardware. Our present architecture still relies on the GRAPE special purpose hardware (not reconfigurable), but next generations will focus on new architectural approaches including custom network and computing architectures. The new hardware is embedded into national and international grid infrastructures.

One of the most important advances needed in global climate models is the development of atmospheric General Circulation Models (GCMs) that can reliably treat convection. Such GCMs require high resolution in local convectively active regions, both in the horizontal and vertical directions. During previous research we have developed an Adaptive Mesh Refinement (AMR) dynamical core that can adapt its grid resolution horizontally.

Our approach utilizes a finite volume numerical representation of the partial differential equations with floating Lagrangian vertical coordinates and requires resolving dynamical processes on small spatial scales. For the latter it uses a newly developed general-purpose library, which facilitates 3D block-structured AMR on spherical grids. The library manages neighbor information as the blocks adapt, and handles the parallel communication and load balancing, freeing the user to concentrate on the scientific modeling aspects of their code. In particular, this library defines and manages adaptive blocks on the sphere, provides user interfaces for interpolation routines and supports the communication and load-balancing aspects for parallel applications. We have successfully tested the library in a 2-D (longitude-latitude) implementation. During the past year, we have extended the library to treat adaptive mesh refinement in the vertical direction. Preliminary results are discussed.

This research project is characterized by an interdisciplinary approach involving atmospheric science, computer science and mathematical/numerical aspects. The work is done in close collaboration between the Atmospheric Science, Computer Science and Aerospace Engineering Departments at the University of Michigan and NOAA GFDL.

The goal of this project is to understand the charge separation mechanisms in biological systems using the molecular orbital theories. Specially, the charge separation in the photosynthetic reaction center is focused on, since the efficiency in use of the solar energy is extraordinary and the reason for it is still kept unknown. Here, a QM/MM theoretical scheme is employed to take the effects of the surrounding proteins onto the pigments into account. To describe such excited electronic structures, a unified theory by MRSCI and DFT is newly invented. For atoms in the MM space, a new sampling method has also been created, based on the statistical physics. By using these theoretical framework, the excited and positively charged states of the special pair, that is, chlorophyll dimmer are planning to be calculated this year.

We present results from a conservative formulation of the spectral element method applied to global atmospheric circulation modeling. Exact local conservation of both mass and energy is obtained via a new compatible formulation of the spectral element method. Compatibility insures that the key integral property of the divergence and gradient operators required to show conservation also hold in discrete form. The spectral element method is used on a cubed-sphere grid to discretize the horizontal directions on the sphere. It can be coupled to any conservative vertical/radial discretization. The accuracy and conservation properties of the method are illustrated using a baroclinic instability test case.

Tracking failures and poor performance across a widely distributed system of resources has proven challenging for many ongoing DOE applications. An example is the Open Science Grid (OSG) project, which currently experiences a roughly 15% job failure rate. This can be an issue not only for Grid computing but for anyone performing large-scale data transfers to remote machines because of the large number of interconnected components and services.

As part of the Center for Enabling Distributed Petascale Science (CEDPS) project we have been building an infrastructure to work with current middleware and existing system tools to more easily track failures and discover anomalous behavior. This consists of a common logging format, the extension of syslog-ng for centralized collection of data, a data summarizer to more easily manage the volume of logging, and an anomaly detection system that can connect to a warning system when unexpected behaviors occur. We are currently working with OSG to deploy a prototype of the full system. The initial logs gathered will be used to extend the analysis tools and to increase the reliability of the services for the SciDAC end user community.

We have developed a simulation capability to model multiscale flow and transport in complex biological systems based on algorithms and software infrastructure developed under the SciDAC APDEC CET. The foundation of this work is a new hybrid fluid-particle method for modeling polymer fluids in irregular microscale geometries that enables long-time simulation of validation experiments. Both continuum viscoelastic and discrete particle representations have been used to model the constitutive behavior of polymer fluids. Complex flow environment geometries are represented on Cartesian grids using an implicit function. Direct simulation of flow in the irregular geometry is then possible using embedded boundary/volume-of-fluid methods without loss of geometric detail. This capability has been used to simulate biological flows in a variety of application geometries including biomedical microdevices, anatomical structures and porous media.

Plasma accelerator methods offer the potential to reduce the size of moderate and high energy accelerators by factors of 1000. In the past few years great advances have been made in the production of low emittance, high quality (i.e., monoenergetic) electron beams with energies between .1 and 1 GeV using ultra-fast (<50 femtoseconds), high power (>10TW) lasers. The most noticeable of these advances were the experimental results presented in the 'Dream Beam' issue of Nature and in a recent issues of Physical Review Letters, Nature, and Nature Physics. The experimental progress have been made due to advances in lasers, diagnostics, plasma sources, and the knowledge of how to control of this highly nonlinear acceleration process. And this experimental progress has occurred simultaneously with and been in part due to advances in modeling capabilities. Using a hierarchy of particle-in-cell (PIC) codes OSIRIS, VORPAL, and QuickPIC, we have performed numerous full scale 3D simulations using parameters quoted from the Nature and Nature Physics articles. Our simulations have predicted results, provided agreement between simulations and experiments (within the shot-to-shot variations of the experiments), and provided insight into the complicated physics of the experiments. Most importantly, as our confidence in the fidelity of our methods increases we can now guide the planning of new experiments, and probe parameters that are not yet available. Thereby providing a 'road map' for generating high quality, high-charge 10 to 100 GeV electron beams for use in high-energy physics and light sources.

The High-Order Method Modeling Environment (HOMME) is a framework to investigate using high-order element-based methods to build conservative and accurate atmospheric general circulation models. Currently, HOMME employs the discontinuous Galerkin and spectral element methods on a cubed-sphere tiled with quadrilateral elements to solve the primitive equations, and has been shown to scale to (10K) processors of a Cray XT 3/4 and (32K) processors of an IBM Blue Gene/L. Here we briefly describe the development of a baroclinic model using the discontinuous Galerkin option in the HOMME framework, present idealized test case results, and provide preliminary performance data.

Recent advances in both computers and algorithms now allow realistic calculations of Quantum Chromodynamics (QCD) interactions using the numerical technique of lattice QCD. The methods used in so-called '2+1 flavor' lattice calculations have been verified both by post-dictions of quantities that were already experimentally well-known and by predictions that occurred before the relevant experimental determinations were sufficiently precise. This suggests that the sources of systematic error in lattice calculations are under control, and that lattice QCD can now be reliably used to calculate those weak matrix elements that cannot be measured experimentally but are necessary to interpret the results of many high-energy physics experiments. These same calculations also allow stringent tests of the Standard Model of particle physics, and may therefore lead to the discovery of new physics in the future.

Quantum Chromodynamics is the theory of nuclear and sub-nuclear physics. It is a celebrated theory and one of its inventors, F. Wilczek, has termed it as '... our most perfect physical theory'. Part of this is related to the fact that QCD can be numerically simulated from first principles using the methods of lattice gauge theory. The computational demands of QCD are enormous and have not only played a role in the history of supercomputers but are also helping define their future. Here I will discuss the intimate relation of QCD and massively parallel supercomputers with focus on the Blue Gene supercomputer and QCD thermodynamics. I will present results on the performance of QCD on the Blue Gene as well as physics simulation results of QCD at temperatures high enough that sub-nuclear matter transitions to a plasma state of elementary particles, the quark gluon plasma. This state of matter is thought to have existed at around 10 microseconds after the big bang. Current heavy ion experiments are in the quest of reproducing it for the first time since then. And numerical simulations of QCD on the Blue Gene systems are calculating the theoretical values of fundamental parameters so that comparisons of experiment and theory can be made.

At its heart, a Type Ia supernova is a problem in turbulent nuclear combustion. There are four sub-problems, each of which has eluded solution for decades, but which can be addressed by large scale simulation. First is the ignition – where and how often the burning is ignited in the convective core of an exploding white dwarf star. The outcome is sensitive to the initial conditions near the star's center, which may be chaotic. Second is the propagation of the flame. Until near the end, the flame is an unresolvably narrow sheet moved around by instabilities and the turbulence that its own motion produces, yet how fast it moves determines the strength and brightness of the explosion. Third is whether and how the subsonic burning makes a spontaneous transition to a detonation. Observations favor this outcome, but the physics of the transition is obscure. Fourth, is the radiation transport problem. Why does the supernova look the way it does and can its light curve be relied upon to do precision cosmology? Our Consortium has made genuine progress in each of these areas, as well as in planning, with observers, future observational strategies for SNAP/JDEM and LSST. A new generation of codes is being optimized for the four tasks.

The Community Atmosphere Model (CAM) is the atmosphere component of the Community Climate System Model (CCSM), and is the largest consumer of computing resources in typical CCSM simulations. The parallel implementation of the Community Atmosphere Model (CAM) employs a number of different domain decompositions. Currently, each decomposition must utilize the same number of MPI processes, limiting the scalability of CAM to that of the least scalable decomposition. This limitation becomes unacceptably restrictive when including additional physical processes such as atmospheric chemistry or cloud-resolving physics. This paper reports on current efforts to improve CAM scalability by allowing the number of active MPI processes to vary between domain decompositions.

We report on some of the interactions between two SciDAC projects: The National Computational Infrastructure for Lattice Gauge Theory (USQCD), and the Performance Engineering Research Institute (PERI). Many modern scientific programs consistently report the need for faster computational resources to maintain global competitiveness. However, as the size and complexity of emerging high end computing (HEC) systems continue to rise, achieving good performance on such systems is becoming ever more challenging. In order to take full advantage of the resources, it is crucial to understand the characteristics of relevant scientific applications and the systems these applications are running on. Using tools developed under PERI and by other performance measurement researchers, we studied the performance of two applications, MILC and Chroma, on several high performance computing systems at DOE laboratories. In the case of Chroma, we discuss how the use of C++ and modern software engineering and programming methods are driving the evolution of performance tools.

Regression analysis, especially the ordinary least squares method which assumes that errors are confined to the dependent variable, has seen a fair share of its applications in aerosol science. The ordinary least squares approach, however, could be problematic due to the fact that atmospheric data often does not lend itself to calling one variable independent and the other dependent. Errors often exist for both measurements. In this work, we examine two regression approaches available to accommodate this situation. They are orthogonal regression and geometric mean regression. Comparisons are made theoretically as well as numerically through an aerosol study examining whether the ratio of organic aerosol to CO would change with age.

Convective astrophysical flows, such as those leading up to Type Ia supernovae, can be characterized by low Mach numbers (M << 1). One can neglect acoustic waves in the modeling of these flows, but must retain compressibility effects due to nuclear reactions and background stratification in order to accurately model the convection and initial burning. We present here a new algorithm, MAESTRO, that is specifically designed for efficient long-time integration leading to ignition. Here, we discuss the state of this new algorithm and its application to Type Ia supernovae.

The FACETS (Framework Application for Core-Edge Transport Simulations) project began in January 2007 with the goal of providing core to wall transport modeling of a tokamak fusion reactor. This involves coupling previously separate computations for the core, edge, and wall regions. Such a coupling is primarily through connection regions of lower dimensionality. The project has started developing a component-based coupling framework to bring together models for each of these regions. In the first year, the core model will be a 1 ½ dimensional model (1D transport across flux surfaces coupled to a 2D equilibrium) with fixed equilibrium. The initial edge model will be the fluid model, UEDGE, but inclusion of kinetic models is planned for the out years. The project also has an embedded Scientific Application Partnership that is examining embedding a full-scale turbulence model for obtaining the crosssurface fluxes into a core transport code.

Edge pedestal height and the accompanying ELM crash are critical elements of ITER physics yet to be understood and predicted through high performance computing. An entirely self-consistent first principles simulation is being pursued as a long term research goal, and the plan is planned for completion in time for ITER operation. However, a proof-of-principle work has already been established using a computational tool that employs the best first principles physics available at the present time. A kinetic edge equilibrium code XGC0, which can simulate the neoclassically dominant pedestal growth from neutral ionization (using a phenomenological residual turbulence diffusion motion superposed upon the neoclassical particle motion) is coupled to an extended MHD code M3D, which can perform the nonlinear ELM crash. The stability boundary of the pedestal is checked by an ideal MHD linear peeling-ballooning code, which has been validated against many experimental data sets for the large scale (type I) ELMs onset boundary. The coupling workflow and scientific results to be enabled by it are described.

Understanding the science behind large-scale simulations and high-throughput experiments requires extracting meaning from data sets of hundreds of terabytes or more. Visualization is the most intuitive means for scientists to understand data at this scale, and the most effective way to communicate their findings with others. Even though visualization technology has matured over the past twenty years, it is still limited by the extent and scale of the data that it can be applied to, and also by the functionalities that were mostly designed for single-user, single-variable, and single-space investigation. The Institute for Ultra-Scale Visualization (IUSV), funded by the DOE SciDAC-2 program, has the mission to advance visualization technologies to enable knowledge discovery and dissemination for peta-scale applications. By working with the SciDAC application projects, Centers for Enabling Technology, and other Institutes, IUSV aims to lead the research innovation that can create new visualization capabilities needed for gleaning insights from data at petascale and beyond to solve forefront scientific problems. This paper outlines what we see as some of the biggest research challenges facing the visualization community, and how we can approach education and outreach to put successful research in the hands of scientists.

Global cloud resolving models at resolutions of 4km or less create significant challenges for simulation output, data storage, data management, and post-simulation analysis and visualization. To support efficient model output as well as data analysis, new methods for IO and data organization must be evaluated. The model we are supporting, the Global Cloud Resolving Model being developed at Colorado State University, uses a geodesic grid. The non-monotonic nature of the grid's coordinate variables requires enhancements to existing data processing tools and community standards for describing and manipulating grids. The resolution, size and extent of the data suggest the need for parallel analysis tools and allow for the possibility of new techniques in data mining, filtering and comparison to observations. We describe the challenges posed by various aspects of data generation, management, and analysis, our work exploring IO strategies for the model, and a preliminary architecture, web portal, and tool enhancements which, when complete, will enable broad community access to the data sets in familiar ways to the community.