Script-Based GPU-Ready ELM Development for Continuous Code Integration ()
1. Introduction
State-of-the-art Earth system models (ESM) provide critical information on climate change. There are several fully-coupled ESMs, including the Energy Exascale Earth System Model (E3SM) that uses code optimized for the US Department of Energy’s (DOE) advanced computers [1] . The E3SM Land Model (ELM) is an integral part of the E3SM framework, simulating the interactions between terrestrial land surfaces and other Earth system components. ELM has been instrumental in understanding hydrologic cycles, biogeophysics, and dynamics of terrestrial ecosystems [2] .
In the development of large-scale, ultrahigh-resolution ELM (uELM), the researchers have implemented the GPU-ready ELM code using OpenACC on Summit at Oak Ridge National Laboratory [3] . To facilitate the automatic porting of individual ELM modules, the researchers have developed a software tool named SPEL [4] . SPEL works within a Functional Unit Testing framework [5] and contains a collection of functions to parse ELM code, generate code segments, analyze dataflow, and support code verification. This paper presents the extension of SPEL functions to support continuous ELM integration based on code change information harvested from Git commits.
2. SPEL Extension to Support Continuous ELM Code Development
ELM uses GitHub to facilitate its full-cycle software development, including design, development, quality assurance, deployment, and maintenance. Currently, SPEL provides functions for ELM code analysis and GPU code generation, as well as optimization (Figure 1). The SPEL software package consists of four major functions. The basic function involves parsing and code generation, initializing ELM, capturing module parameters, and generating Fortran code. The unit test creation function is responsible for creating a unit test driver, managing data input and output, and executing code. The GPU porting function handles GPU data regions, offloading to the GPU, and performance tuning. Lastly, the verification function is used for verifying testing results.
Our objective is to expand SPEL’s capabilities to support continuous integration of ELM code. To achieve this, we have developed a script-based function that follows a two-step workflow. The first step involves analyzing and processing information from the Git commit output of the CPU code base. The second step involves calling SPEL’s functions to generate GPU-ready code for new modifications. This function skips all unchanged OpenACC segments, examines the structure and scope of code modifications, and generates code with appropriate directives (clauses) for these altered sections. The workflow of using SPEL GitUtil is also shown in Figure 1.
The ELM code can undergo three major types of change:
1) Change to land subgrid datatypes (such as the addition of topographic units) and associated ELM processes. This modification impacts many ELM processes, including Canpoy FLux, Ecosystem Dynamics, and Soil Temperature.
2) Change to ELM process parameters (such as the addition of parameters for snowmelt) and associated functions. This type of change requires adjustments in
![]()
Figure 1. SPEL functions used for uELM code porting within a Functional Unit Testing framework.
GPU data region creation and ELM function kernels and impacts limited ELM processes.
3) Change to individual ELM functions (such as subroutines). This type of change results in data and code changes in specific subroutines. The third type of change is considered as the basic ELM code change.
3. Code Generation for Modifications in a Single ELM Source File
This paper presents a code generation procedure for VerticalProfileMod.F90 that holds routines for the vertical discretization of C and N inputs into decomposing pools. It is an example of the basic ELM code change without new data (variable) introduced. There are four steps:
1) Find the change history of the module (i.e., git log—follow—VerticalProfileMod.F90).
2) Select a specific commit (i.e., 067be7cbc99cd349770d4eeb827-0b58f47498dfe, committed by Michael A. Brunke on June 10, 2020, to add “profile adjustments w/ variable soil in VerticalProfileMod”).
3) Extract the modified file (i.e. git show {commit}) and save the Git change logs. Four code sections in the VerticalProfileMod.F90 were changed, annotated by special symbols “@@” (Table 1).
4） Develop a new SPEL function (gitutils) use regex to analyze the Git output file (i.e., list of filenames, changed sections), mark the region of code change in the existing GPU code, remove all the OpenACC directives in the region, modify the code (based on Git commit), then call SPEL functions to generate GPU-ready code only for the changed sections.
For illustration purpose, the first code change section (starting at line 141) (left) and the newly generated GPU-ready code (right) is shown in Figure 2. Gitutils first parses the code change section and determines the impacted region of the code change (the outer loop that beyond the texts in the commit log), secondly removes existing acc statements in the target region (first two acc statements), and then changes the code (the old 11-line code became a new 22-line code with new statements (cyan)), finally uses appropriate SPEL functions to regenerate the GPU code (add three acc statements (gray) at the right places).
![]()
Table 1. Changed code sections in VerticalProfileMod.F90.
![]()
Figure 2. The input Git commit log (left) and the new GPU-ready code (right) for the first changed code section in VerticalProfileMod.F90.
4. Conclusion
This study successfully extended the SPEL functions to generate GPU-ready code for continuous ELM integration. The exemplar case demonstrated the effectiveness of the GPU-code generation procedure based on the GitHub change logs. This study laid the foundation for more complicated cases via iterative approaches, which will be reported in future publications.
Acknowledgements
This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, funded by the U.S. Department of Energy, Office of Biological and Environmental Research. This research used resources from the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory.
NOTES
*Both authors contributed equally.