ORNL CAAR: Preparing PIConGPU, a Plasma Physics application for Frontier Exascale system

PI: Sunita Chandrasekaran

UDEL Students: Vineeth Gutta

Collaborators from HZDR/CASUS: Michael Bussmann, Alexander Debus, Guido Juckeland, Rene Widera, Sergei Bastrakov, Thomas Kluge, Klaus Steiniger, Jeff Kelling, Richard Pausch, Marco Garten, Felix Meyer, Matthias Werner, Franz Poeschel, Jeyhun Rustamov

Funding Agency and Period: CASUS – Center for Advanced Systems Understanding, Helmholtz-Zentrum Dresden-Rossendorf e.V. (HZDR) Untermarkt 20, D-02826 Görlitz, Germany, May 2020 to March 2022

Project Duration: 10/01/2019 - present

Project Summary:

This is a prestigious software project, called Center for Accelerated Application Readiness (CAAR), awarded by ORNL in September 2019. Only 8 CAAR teams have been selected by ORNL across the United States to work on software stacks for one of the upcoming Exascale systems, Frontier to be housed at ORNL by 2021 timeframe.

The CAAR project uses a Particle In Cell GPU (PIConGPU), an extremely scalable, heterogeneous, fully relativis- tic C++ code. PIConGPU is available as open-source project on GitHub, for plasma and laser-plasma physics used to develop advanced particle accelerators for radiation therapy of cancer, high energy physics and photon science. The code is suitable for production-quality runs on state-of-the-art supercomputers driven by manycore accelerators as well as traditional architectures with a single, maintainable code base.

With Frontier, we believe the complex plasma dynamics that govern the final particle beam properties can now be studied at an unprecedented temporal and spatial resolution. This will provide accelerator development with the high quality data needed to advance plasma accelerators towards application.

PIConGPU uses the ALPAKA software stack to target almost all many-core compute platforms currently on the market using a single source, redundant parallel hierarchy approach and the openPMD ecosystem for high performant I/O. Alpaka offers a template-based unified C++14 programming model for multicore and manycore architectures, namely CPUs (Intel, AMD, ARM, POWER) and GPUs (Nvidia, AMD) by leveraging the APIs OpenMP, TBB, Fibers, CUDA as well as an initial version of HIP.

The challenges in this project include creating a working and a stable OpenMP 4.5/5.x and an OpenACC backend for Alpaka in order to target Frontier that will comprise of AMD CPUs and GPUs. PIConGPU will be one of the 8 CAAR codes (amidst other ECP codes) that will will have early access to Frontier in order to study the software and hardware limitations before Frontier is made available for other real-world applications.