.. _platforms:

Running on different platforms
======================================

HPC platforms
'''''''''''''

Below is a list of platforms where NorESM2 has been installed, including platform specific intructions. 

**<noresm-base>**: the name of the folder where the model code has been downloaded (cloned from git)

The configurations files with the platform specific settings are found in ::
  
   <noresm-base>/cime/config/cesm/machines/config_machines.xml
   <noresm-base>/cime/config/cesm/machines/config_compilers.xml
   <noresm-base>/cime/config/cesm/machines/config_batch.xml
   
    
Usually there is no need to change these files.

Fram @ Sigma2
^^^^^^^^^^^^^
Configuration files for running NorESM2 on Fram are distributed in all branches of the noresm code.

Input data is stored in /cluster/shared/noresm/inputdata/

Apply for membership in NorESM shared data storage (manager: mben@norceresearch.no) for access to the folder.

The run and archive directories are stored in /cluster/work/users/<user_name>/

Create a new case: ::

    ./create_newcase --case ../../../cases/<casename> --mach fram --res <resolution> --compset <compset_name> --project <project_name> --user-mods-dir <user_mods_dir> --run-unsupported


Betzy @ Sigma2
^^^^^^^^^^^^^^^
NorESM2 working on BETZY with CIME tag cime5.6.10_cesm2_1_rel_06-Nor_v1.0.4

Configuration files for running NorESM2 on Betzy are currently being put in all branches of the noresm code.

Input data is stored in /cluster/shared/noresm/inputdata/

Apply for membership in NorESM shared data storage (manager: mben@norceresearch.no) for access to the folder.

The run and archive directories are stored /cluster/work/users/<user_name>/

NorESM requires at least Python3 for setup and building. A recommended set of modules is provided by
::

  module purge
  module load GCCcore/11.3.0
  module load git/2.36.0-GCCcore-11.3.0-nodocs
  module load Python/3.10.4-GCCcore-11.3.0

Create a new case: ::

    ./create_newcase --case ../../../cases/<casename> --mach betzy --res <resolution> --compset <compset_name> --project <project_name> --user-mods-dir <user_mods_dir> --run-unsupported

Setting number of nodes on Betzy
---------------------------------
In fully coupled setup, for grid f19_tn14 and f09_tn14, various processors configurations are added and can be used by setting ``--pecount`` = S, L, M or X1
when creating a new case.

For NorESM2-MM, using grid f09_tn14, ``--pecount`` will give the following number of nodes: ::

    S = 4
    M = 9
    L = 15
    X1 = 17

and for NorESM2-LM, using grid f19_tn14, ``--pecount`` it will give the following number of nodes: ::

    S = 4
    M = 8
    X1 = 10

L does not exists for f19_tn14.

E.g. creating a new case running on 4 nodes ::

    ./create_newcase –case ../../../cases/<casename> --mach betzy –-res <resolution> --compset <compset_name> --project <project_name> --pecount= S --user-mods-dir <user_mods_dir> --run-unsupported  
    
 
.. note::
  Please note that these Betzy settings are included in tag noresm2.0.4 and subsequent tags

Queue options on Fram and on Betzy
-----------------------------------
On fram there are different queues for testing and development experiments (usually short runs on few nodes) and longer experiments. If you want to run simulations using different queue options than *normal*, in env_batch.xml:

Add to the directives 
::

    <directive> --qos=devel</directive>


for the development queue and change (only on Fram) the node settings to: 
::

   <queue walltimemax="00:30:00" nodemin="1" nodemax="4" default="true">devel</queue>


For the short (Fram) and preproc (Betzy) queue options, add
::

    <directive> --qos=short</directive>
    
    or 
    
    directive> --qos=preproc</directive>


for the development queue and change (only on Fram) the node settings to: 
::

 <queue walltimemax="02:00:00" nodemin="1" nodemax="10" default="true">short</queue>
  

  
for the short/preproc queue. 

| For a detailed guide on how to set up, submit and choosing the right job see: 
| https://documentation.sigma2.no/jobs/submitting.html  
| https://documentation.sigma2.no/jobs/choosing_job_types.html  
| 

Nebula @ NSC
^^^^^^^^^^^^
Configuration files for running NorESM2 on Nebula are distributed in the release tags release-noresm2* and in the noresm2 origin/noresm2 branch. If Nebula configurations are missing in your copy of the model, the files can be found in the following folder on Nebula:
::

  /nobackup/forsk/noresm/nebula_config_noresm2/
    

Apply for membership in NorESM shared data storage (manager: adag@met.no) for access to the folder.

Copy the files in the above folder to:
::

    cd <noresm-base>/cime/config/cesm/machines/
    cp /nobackup/forsk/noresm/nebula_config_noresm2/* .


Input data is stored in ``/nobackup/forsk/noresm/inputdata/``

The run and archive directories are stored ``/nobackup/forsk/<user_name>/``

Before configuring and compiling the model, add ``export LMOD_QUIET=1`` to your .bashrc
::

    cd
    vi .bashrc


Create a new case:
::

    ./create_newcase --case ../../../cases/<casename> --mach nebula --res <resolution> --compset <compset_name> --project <project_name> --user-mods-dir <user_mods_dir> --run-unsupported


Tetralith @ NSC
^^^^^^^^^^^^^^^

Configuration files for running NorESM2 on Tetralith are distributed e.g. in the tag release-noresm2.0.7. If Tetralith configurations are missing in your copy of the model the files can be found in the following folder on Tetralith:
::

  /proj/cesm_input-data/tetralith_config_noresm2
    

Apply for membership in CESM climate model shared data storage (NAISS 2024/23-95) for access to the folder.

Copy the files in the above folder to if configuration files are not distributed with the code:
::

    cd <noresm-base>/cime/config/cesm/machines/
    cp /proj/cesm_input-data/tetralith_config_noresm2/* .


Input data is stored /proj/cesm_input-data/ 

Before configuring and compiling the model, clear your environment and load the following modules:
::

  module purge 
  module load buildenv-intel/2023a-eb
  module load netCDF-HDF5/4.9.2-1.12.2-hpc1
  module load Perl/5.38.0-hpc1-gcc-system



Create a new case:

::

    ./create_newcase --case ../../../cases/<casename> --mach triolith --res <resolution> --compset <compset_name> --pecount M --ccsm_out <NorESM_ouput_folder>



Virtual Machine with Conda (@ https://www.nrec.no/ for example)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

This section describes how to install all the software environment (including compilers and libraries) needed to run CESM/NorESM on a Virtual Machine (like those available on the Norwegian Research and Education Cloud, the Google Cloud Platform, etc.), but a similar process allows to run the model(s) on a personal computer, laptop or desktop running **Centos7** (this distribution is convenient to use since it already contains most of the essential software packages).

The objective here is not to compete against HPCs in terms of sheer computing power, but to satisfy the everyday needs of the vast majority of CESM/NorESM developpers in terms of model development, debugging or testing, as well as for training/teaching purposes.

For this example we start with a completely empty machine with the Centos7 Linux Distribution, 16x Intel Core Processors (Haswell, no TSX), 128GB RAM, and a 100GiB volume (disk) attached on **/dev/sdb**.

The name of the user is **centos** (if your user name is different you will have to use your *username* instead).

The first step is to format the volume (if your disk is already formated and/or contains data, skip this step, but still create the **/opt/uio** folder since this is where the model(s) are configured to read/write).
::

  sudo mkfs.ext4 /dev/sdb


then mount it at /opt/uio
::

  sudo mkdir /opt/uio

  sudo chown -R centos /opt/uio

  sudo chgrp -R centos /opt/uio

  sudo mount /dev/sdb /opt/uio

  cd /opt/uio


and create the following folders:
::

  mkdir /opt/uio/inputdata

  mkdir /opt/uio/work

  mkdir /opt/uio/archive

  mkdir /opt/uio/archive/cases



Now we can install a few packages which will be needed later (to get the model(s), etc.) and miniconda (accept the terms of the license and accept the default location **/home/centos/miniconda3**, then answer yes to the question *"Do you wish the installer to initialize Miniconda3 by running conda init"*, exit the virtual machine and re-login).
::

  sudo yum install wget git subversion csh -y

  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

  bash Miniconda3-latest-Linux-x86_64.sh

  exit 


You will notice the next time you login the Virtual Machine that the prompt starts with *(base)* which indicates that you are in the base conda environment (since you accepted it during the miniconda install).

We recommend to create a new **esm** conda environment before adding the **bioconda** and **conda-forge** channels (in this order) and installing cesm
::

  conda create -n esm

  conda activate esm

  conda config --add channels bioconda

  conda config --add channels conda-forge

  conda install cesm=2.1.3 


The prompt should start with *(esm)* indicating that the esm conda environment has been activated, and every time you login you will have to type **conda activate esm** to be able to run the model(s).

This will have installed CESM2.1.3 as well as all the necessary compilers and libraries (HDF5, NetCDF, MKL, etc.) and their dependencies, and the very same environment can be used with NorESM.

In order to run the model(s) you still need configuration files (namely *config*, *config_machines.xml* and *config_compilers.xml*). These will eventually come with NorESM, but for the sake of convenience we provide hereafter an example of such files which have to be located in a **.cime** folder in your home directory (simply copy & past the content of the following cell to generate the files automatically and be carefull not to add any odd characters or lines since CESM/NorESM are extremely picky about it).

Notice that you only need to do this once, since both CESM and NorESM will use these configurations, and that the name of the machine created is **espresso**. 
::

  cd /home/centos

  mkdir .cime

  cd .cime

  cat >> config << EOF
  [main]
  CIME_MODEL=cesm
  EOF

  cat >> config_machines.xml << EOF
  <?xml version="1.0"?>
  <config_machines>
    <machine MACH="espresso">
      <DESC> Virtual Machine with 16 VCPUs and 128GiB memory
             OS is Centos7, Conda CESM environment
      </DESC>
      <NODENAME_REGEX>UNSET</NODENAME_REGEX>
      <OS>LINUX</OS>
      <PROXY>UNSET</PROXY>
      <COMPILERS>gnu</COMPILERS>
      <MPILIBS>mpich</MPILIBS>
      <SAVE_TIMING_DIR>UNSET</SAVE_TIMING_DIR>
      <CIME_OUTPUT_ROOT>/opt/uio/work</CIME_OUTPUT_ROOT>
      <DIN_LOC_ROOT>/opt/uio/inputdata</DIN_LOC_ROOT>
      <DIN_LOC_ROOT_CLMFORC>UNSET</DIN_LOC_ROOT_CLMFORC>
      <DOUT_S_ROOT>/opt/uio/archive/$CASE</DOUT_S_ROOT>
      <BASELINE_ROOT>UNSET</BASELINE_ROOT>
      <CCSM_CPRNC>UNSET</CCSM_CPRNC>
      <GMAKE_J>16</GMAKE_J>
      <BATCH_SYSTEM>none</BATCH_SYSTEM>
      <SUPPORTED_BY>noresmCommunity</SUPPORTED_BY>
      <MAX_TASKS_PER_NODE>16</MAX_TASKS_PER_NODE>
      <MAX_MPITASKS_PER_NODE>16</MAX_MPITASKS_PER_NODE>
      <PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
      <mpirun mpilib="default">
        <executable>mpiexec</executable>
        <arguments>
          <arg name="anum_tasks"> -np \$TOTALPES</arg>
        </arguments>
      </mpirun>
      <module_system type="none"/>
      <environment_variables>
        <env name="KMP_STACKSIZE">64M</env>
      </environment_variables>
      <resource_limits>
        <resource name="RLIMIT_STACK">-1</resource>
      </resource_limits>
    </machine>
  </config_machines>
  EOF

  cat >> config_compilers.xml << EOF
  <?xml version="1.0"?>
  <config_compilers version="2.0">
    <compiler MACH="espresso">
      <LD>mpifort</LD>
      <AR>x86_64-conda_cos6-linux-gnu-ar</AR>
      <SFC>x86_64-conda_cos6-linux-gnu-gfortran</SFC>
      <SCC>x86_64-conda_cos6-linux-gnu-cc</SCC>
      <SCXX>x86_64-conda_cos6-linux-gnu-c++</SCXX>
      <MPIFC>mpifort</MPIFC>
      <MPICC>mpicc</MPICC>
      <MPICXX>mpicxx</MPICXX>
      <NETCDF_PATH>/home/centos/miniconda3/envs/esm</NETCDF_PATH>
      <FFLAGS>
        <append DEBUG="FALSE"> -O2 </append>
        <append MODEL="blom"> -fdefault-real-8 </append>
        <append MODEL="cam"> -finit-local-zero </append>
      </FFLAGS>
      <SLIBS>
        <append> -L\$(NETCDF_PATH)/lib -lnetcdff -lnetcdf -ldl </append>
        <append> -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lomp -lpthread -lm </append>
      </SLIBS>
    </compiler>
  </config_compilers>
  EOF

To create a new CESM case F2000climo at resolution f19_g17 and run it for **1 day**, and because (for CESM only) *create_newcase* has been added to the *PATH*, simply type (from anywhere on the machine):
::

  create_newcase --case /opt/uio/archive/cases/conda_CESM213_F2000climo_f19_g17 --compset F2000climo --res f19_g17 --machine espresso --run-unsupported

  cd /opt/uio/archive/cases/conda_CESM213_F2000climo_f19_g17

  NUMNODES=-1

  ./xmlchange --file env_mach_pes.xml --id NTASKS --val ${NUMNODES}

  ./xmlchange --file env_mach_pes.xml --id NTASKS_ESP --val 1

  ./xmlchange --file env_mach_pes.xml --id ROOTPE --val 0

  ./xmlchange STOP_N=1

  ./xmlchange STOP_OPTION=ndays

  ./case.setup

  ./case.build

  ./case.submit


Hopefully this should create the case, configure it, compile it (for this particular machine the compilation time is less then 3 minutes) and run it (starting with the download of the necessary input files the first time you run it).

For NorESM, first clone the github repository, here in /opt/uio/**noresm2**, as follows (be careful: you have to be in the **(base)** conda environment for that):
::

  cd /opt/uio

  git clone -b noresm2 https://github.com/NorESMhub/NorESM.git noresm2

  cd noresm2/

  sed -i.bak "s/'checkout'/'checkout', '--trust-server-cert'/" ./manage_externals/manic/repository_svn.py
 
  ./manage_externals/checkout_externals -v 


To create a "similar" NorESM case NF2000climo at resolution f19_tn14 and also run it for **1 day**, and after having activated the **(esm)** environment (if you are not already in it), do:
::

  cd /opt/uio/noresm2/cime/scripts

  ./create_newcase --case /opt/uio/archive/cases/conda_NorESM_NF2000climo_f19_tn14 --compset NF2000climo --res f19_tn14 --machine espresso --run-unsupported

  cd /opt/uio/archive/cases/conda_NorESM_NF2000climo_f19_tn14

  NUMNODES=-1

  ./xmlchange --file env_mach_pes.xml --id NTASKS --val ${NUMNODES}

  ./xmlchange --file env_mach_pes.xml --id NTASKS_ESP --val 1

  ./xmlchange --file env_mach_pes.xml --id ROOTPE --val 0

  ./xmlchange STOP_N=1

  ./xmlchange STOP_OPTION=ndays

  ./case.setup

  ./case.build

  ./case.submit


On our machine the compilation takes less then 3 minutes, and if everything went well the input files should download automatically before the run starts.

Note: AeroTab is supposed to be a folder, if a file has been created instead simply add a "trailing slash" (/) at line 1946 in  ``components/cam/bld/namelist_files/namelist_defaults_cam.xml`` (which has to be written as: <aerotab_table_dir>noresm-only/atm/cam/camoslo/AeroTab_8jun17/</aerotab_table_dir>) and resubmit.

Adding a new platform
'''''''''''''''''''''

Edit the following files:
::

  config_batch.xml  
  config_compilers.xml  
  config_machines.xml

located in

::

  <noresm-base>/cime/config/cesm/machines/


config_batch.xml
^^^^^^^^^^^^^^^^

Add a batch_system entry in this file for your platform with appropriate settings. See examples below.

Machine example with SLURM batch system

on Fram:
::

  <batch_system MACH="fram" type="slurm">
    <batch_submit>sbatch</batch_submit>
    <submit_args>
      <arg flag="--time" name="$JOB_WALLCLOCK_TIME"/>
      <arg flag="-p" name="$JOB_QUEUE"/>
      <arg flag="--account" name="$PROJECT"/>
    </submit_args>
    <directives>
      <directive> --ntasks={{ total_tasks }}</directive>
      <directive> --export=ALL</directive>
      <directive> --switches=1</directive>
    </directives>
    <queues>
      <queue walltimemax="00:59:00" nodemin="1" nodemax="288" default="true">normal</queue>
    </queues>
  </batch_system>


On Tetralith:
::

  <batch_system type="slurm" MACH="tetralith">
    <batch_submit>sbatch</batch_submit>
    <submit_args>
      <arg flag="--time" name="$JOB_WALLCLOCK_TIME"/>
      <arg flag="--account" name="$PROJECT"/>
    </submit_args>
    <queues>
      <queue walltimemax="168:00:00" nodemin="1" default="true">default</queue>
      <queue walltimemax="01:00:00" nodemin="1" nodemax="4" >development</queue>
    </queues>
  </batch_system>


Machine example with PBS batch system
::

  <batch_system MACH="vilje" type="pbs">
    <submit_args>
      <arg flag="-N cesmRun"/>
    </submit_args>
    <directives>
      <directive>-A nn2345k</directive>
      <directive>-l select={{ num_nodes }}:ncpus={{ MAX_TASKS_PER_NODE }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}</directive>
    </directives>
    <queues>
      <queue walltimemax="00:59:00" nodemin="1" nodemax="9999" default="true">workq</queue>
    </queues>
    <!--walltimes>
                            <walltime default="true">00:59:00</walltime>
    </walltimes-->
  </batch_system>


config_compilers.xml
^^^^^^^^^^^^^^^^^^^^
 
Add a compiler entry in this file for your platform with appropriate settings. See examples below.

On Fram:
::

   <compiler MACH="fram">
     <CPPDEFS>
       <append> -D$(OS) </append>
     </CPPDEFS>
     <FFLAGS>
       <append> -xCORE-AVX2 -no-fma </append>
     </FFLAGS>
     <NETCDF_PATH>$(EBROOTNETCDFMINFORTRAN)</NETCDF_PATH>
     <PNETCDF_PATH>$(EBROOTPNETCDF)</PNETCDF_PATH>
     <MPI_PATH>$(MPI_ROOT)</MPI_PATH>
     <MPI_LIB_NAME>mpi</MPI_LIB_NAME>
     <FFLAGS>
       <append DEBUG="FALSE"> -O2 </append>
       <append MODEL="blom"> -r8 </append>
       <append MODEL="cam"> -init=zero,arrays </append>
     </FFLAGS>
     <MPICC> mpiicc </MPICC>
     <MPICXX> mpiicpc </MPICXX>
     <MPIFC> mpiifort </MPIFC>
     <PIO_FILESYSTEM_HINTS>lustre</PIO_FILESYSTEM_HINTS>
     <SLIBS>
       <append>-mkl=sequential -lnetcdff -lnetcdf</append>
     </SLIBS>
  </compiler>


On Tetralith:
::
 
   <compiler MACH="tetralith" COMPILER="intel">
    <MPICC> mpiicc  </MPICC>
    <MPICXX> mpiicpc </MPICXX>
    <MPIFC> mpiifort </MPIFC>
    <PNETCDF_PATH>$ENV{PNETCDF_DIR}</PNETCDF_PATH>
    <NETCDF_PATH>$ENV{NETCDF_DIR}</NETCDF_PATH>
    <SLIBS>
      <append>-L$(NETCDF_PATH)/lib -lnetcdf -lnetcdff</append>
    </SLIBS>
    <FFLAGS>
      <append> -xHost -fPIC -mcmodel=large </append>
    </FFLAGS>
    <FFLAGS>
      <append DEBUG="FALSE"> -O0 -xAVX </append>
      <append MODEL="blom"> -r8 </append>
    </FFLAGS>
    <CFLAGS>
      <append> -xHost -fPIC -mcmodel=large </append>
    </CFLAGS>
    <LDFLAGS>
      <append> -mkl </append>
    </LDFLAGS>
  </compiler>
 
config_machines.xml
^^^^^^^^^^^^^^^^^^^
 
Add a machine entry in this file for your platform with appropriate settings. See examples below.

On Fram:
::

  <machine MACH="fram">
    <DESC>Lenovo NeXtScale M5, 32-way nodes, dual 16-core Xeon E5-2683@2.10GHz, 64 GiB per node, os is Linux, batch system       is SLURM</DESC>
    <OS>LINUX</OS>
    <COMPILERS>intel</COMPILERS>
    <MPILIBS>impi</MPILIBS>
    <CIME_OUTPUT_ROOT>/cluster/work/users/$USER/noresm</CIME_OUTPUT_ROOT>
    <DIN_LOC_ROOT>/cluster/shared/noresm/inputdata</DIN_LOC_ROOT>
    <DIN_LOC_ROOT_CLMFORC>UNSET</DIN_LOC_ROOT_CLMFORC>
    <DOUT_S_ROOT>/cluster/work/users/$USER/archive/$CASE</DOUT_S_ROOT>
    <DOUT_L_ROOT>/projects/NS2345K/noresm/cases</DOUT_L_ROOT>
    <DOUT_L_HOSTNAME>login.nird.sigma2.no</DOUT_L_HOSTNAME>
    <!--DOUT_L_MSROOT>UNSET</DOUT_L_MSROOT-->
    <BASELINE_ROOT>UNSET</BASELINE_ROOT>
    <CCSM_CPRNC>UNSET</CCSM_CPRNC>
    <GMAKE_J>8</GMAKE_J>
    <BATCH_SYSTEM>slurm</BATCH_SYSTEM>
    <SUPPORTED_BY>noresmCommunity</SUPPORTED_BY>
    <MAX_TASKS_PER_NODE>32</MAX_TASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE>32</MAX_MPITASKS_PER_NODE>
    <PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
    <mpirun mpilib="mpi-serial">
      <executable></executable>
    </mpirun>
    <mpirun mpilib="default">
      <executable>mpirun</executable>
    </mpirun>
    <module_system type="module">
      <init_path lang="perl">/cluster/software/lmod/lmod/init/perl</init_path>
      <init_path lang="python">/cluster/software/lmod/lmod/init/env_modules_python.py</init_path>
      <init_path lang="csh">/cluster/software/lmod/lmod/init/csh</init_path>
      <init_path lang="sh">/cluster/software/lmod/lmod/init/sh</init_path>
      <cmd_path lang="perl">/cluster/software/lmod/lmod/libexec/lmod perl</cmd_path>
      <cmd_path lang="python">/cluster/software/lmod/lmod/libexec/lmod python</cmd_path>
      <cmd_path lang="sh">module</cmd_path>
      <cmd_path lang="csh">module</cmd_path>
      <modules>
        <command name="purge">--force</command>
        <command name="load">StdEnv</command>
        <!-- djlo Deactivated THT settings -->
        <!--command name="load">intel/2016a</command-->
        <!--command name="load">netCDF-Fortran/4.4.3-intel-2016a</command-->
        <!--command name="load">PnetCDF/1.8.1-intel-2016a</command-->
        <!--command name="load">CMake/3.5.2-intel-2016a</command-->
        <command name="load">intel/2018a</command>
        <command name="load">netCDF-Fortran/4.4.4-intel-2018a-HDF5-1.8.19</command>
        <command name="load">PnetCDF/1.8.1-intel-2018a</command>
        <command name="load">CMake/3.9.1</command>
      </modules>
    </module_system>
    <environment_variables>
      <env name="KMP_STACKSIZE">64M</env>
      <env name="I_MPI_EXTRA_FILESYSTEM_LIST">lustre</env>
      <env name="I_MPI_EXTRA_FILESYSTEM">on</env>
    </environment_variables>
    <resource_limits>
      <resource name="RLIMIT_STACK">-1</resource>
    </resource_limits>
  </machine>

On Tetralith:
::
 
   <machine MACH="tetralith">
    <DESC>Tetralith Linux Cluster (NSC, Sweden), 32 pes/node, batch system SLURM</DESC>
    <OS>LINUX</OS>
    <COMPILERS>intel</COMPILERS>
    <MPILIBS>impi</MPILIBS>
    <PROJECT>snic2019-1-2</PROJECT>
    <CHARGE_ACCOUNT>bolinc</CHARGE_ACCOUNT>
    <CIME_OUTPUT_ROOT>/proj/$CHARGE_ACCOUNT/users/$ENV{USER}/noresm2</CIME_OUTPUT_ROOT>
    <DIN_LOC_ROOT>/proj/cesm_input-data/inputdata/</DIN_LOC_ROOT>
    <DIN_LOC_ROOT_CLMFORC>/proj/cesm_input-data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
    <DOUT_S_ROOT>$CIME_OUTPUT_ROOT/cesm_archive/$CASE</DOUT_S_ROOT>
    <BASELINE_ROOT>$CIME_OUTPUT_ROOT/cesm_baselines</BASELINE_ROOT>
    <CCSM_CPRNC>/$CIME_OUTPUT_ROOT/cesm_tools/cprnc/cprnc</CCSM_CPRNC>
    <GMAKE_J>4</GMAKE_J>
    <BATCH_SYSTEM>slurm</BATCH_SYSTEM>
    <SUPPORTED_BY>snic</SUPPORTED_BY>
    <MAX_TASKS_PER_NODE>32</MAX_TASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE>32</MAX_MPITASKS_PER_NODE>
    <PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
    <mpirun mpilib="default">
      <executable>mpprun</executable>
    </mpirun>
    <module_system type="none">
    </module_system>
  </machine>