Lifetime Tracer Simulation: Difference between revisions

From icon-art guide
Jump to navigation Jump to search
No edit summary
(Adding Description "Setting up the Runscript")
Line 2: Line 2:


In this example it is shown how to simulate a lifetime driven tracer with simplified chemistry in ICON-ART. This tutorial teaches you...
In this example it is shown how to simulate a lifetime driven tracer with simplified chemistry in ICON-ART. This tutorial teaches you...
* the setup of the runscript
* the correct setup of the xml-file for such tracers.
* the correct setup of the xml-file for such tracers.
* the structure of lifetime tracers in xml-files
* the structure of lifetime tracers in xml-files
* editing output variables
* editing output variables


== Configuration case ==
The depicted case is dealing with biogenic very short-lived species (VSLS) with a very short lifetime, more accurately Bromoform (<chem>CHBr3</chem>) and Dibromomethane (<chem>CH2Br2</chem>). Like of most VSLSs the major source of <chem>CHBr3</chem> and <chem>CH2Br2</chem> is the ocean which leads too a large gradient with increasing height in the concentration of these tracers. Bromoform is mainly depleted by photolysis in the Troposphere whereas Dibromomethanes main loss is due to Hydroxylradicals (OH). To assess the ability to simulate the transport of VSLS from the surface to the lower stratosphere, this case study uses an idealized chemical tracer approach.

The simulation is modeling the 01 October 2012 and is initialized with data from the ECMWF Integrated Forecast System (IFS) and includes boundary conditions and chemical lifetimes from the WMO Ozone assessment 2010. The boundary conditions and the chemical lifetimes are recalculated in a sort of rate at which the substances are depleted from the atmosphere with help of the implicit solution of the balance equation:
<math>\frac{\partial(\bar{\rho)\hat{\Psi_{l}}/{\partial t} = -\nabla\cdot (\hat{v}\bar{\rho}\hat{\Psi_{l}})- \nabla\cdot\bar{(\rho v''\Psi_{g,l}''}+P_l-L_l+E_l</math>

Here <math>\rho</math> is the density of air, <math>\hat{\Psi_{l}}</math> the barycentric-averaged mass mixing ratio, <math>\nabla\cdot (\hat{v}\bar{\rho}\hat{\Psi_{l}})</math> indicates the flux divergence that includes the horizontal and vertical advection of the gaseous compound l and <math>\nabla\cdot\bar{(\rho v''\Psi_{l}'')}</math> indicates the change due to turbulent fluxes. Further <math>P_l</math> describes the production rate due to chemical reactions, <math>L_l</math> the respective loss rate and emissions are noted with <math>E_l</math>. Everything is related to the respective compound l.

In this example no emission data is used.


== Setting up the Runscript ==

Let's start with the runscript that has to be prepared. Please note that the in the following explained parts have to be printed in one runscript-file with the naming designation "xyz.run". Here it is named <code>lifetime_tracer_test.run</code> but of course you can call it differently as well.

If you've also worked through the example of the [[Simplified Chemistry]], you can use nearly the same runscript. Note that you have to change the paths from Part 1, the timing settings from Part 2, the output variables from Part 3, the emission settings as well as the path of the chemtracer-xml-file in the ART-settings from Part 4 and finally the timing in the job settings from Part 5. Details can be found below.

Inside of that, first check that all your directories are correct, probably they have to be adjusted. Abbreviations used here are the following:
*CENTER: Your organization
*EXPNAME: name of your ICON-Simulation
*OUTDIR: Directory where the simulation output will be stored
*ARTFOLDER: Directory where the ICON-ART code is stored
*INDIR: Directory where the necessary Input data are stored
*EXP:
*lart: For ICON-ART Simulation that has to be switched to <code>Oheim_simple_icon.run</code>.

<div class="toccolours mw-collapsible mw-collapsed">
Part 1: Runscript Directory Settings (Example configuration)
<pre class="mw-collapsible-content">
#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-liftime_tracer_test
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------
</pre>
</div>

Additionally in the next lines of code you set the timing. In this simulation we only simulate one day (01 October 2012). To really catch all day times and so the time dependent solar radiation, the output interval is set to 1 hours to calculate <chem>CHBr3</chem> and <chem>CH2Br2</chem> to every time of the day. Because of the Photolysis dependency of <chem>CHBr3</chem> this is particularly important.
<div class="toccolours mw-collapsible mw-collapsed">
Part 2: Runscript Timing Settings (Example configuration)
<pre class="mw-collapsible-content">
! run_nml: general switches ----------
&amp;# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-02T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-02T00:00:00Z"}
output_interval="PT1H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"
</pre>
</div>

Further, all the namelist parameters (from the regular ICON model without ART-extension) have to be set. For a regular ICON-ART-Simulation the following settings are recommended - if not stated differently. For a detailed description, check out the ICON Documentation ([https://code.mpimet.mpg.de/attachments/download/19568/ICON_tutorial_2019.pdf Drill et. al. (2019)]).

<div class="toccolours mw-collapsible mw-collapsed">
Part 3: Runscript ICON-Parameter and -Namelist Settings (Example configuration)
<pre class="mw-collapsible-content">
# model parameters
model_equations=3 # equation system
# 1=hydrost. atm. T
# 1=hydrost. atm. theta dp
# 3=non-hydrost. atm.,
# 0=shallow water model
# -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
lRestart = .false.
/
&master_time_control_nml
experimentStartDate = "$start_date"
experimentStopDate = "$end_date"
forecastLeadTime = "$leadtime"
checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
model_type=1
model_name="ATMO"
model_namelist_filename="$atmo_namelist"
model_min_rank=1
model_max_rank=65536
model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
nproma = 8 ! optimal setting 8 for CRAY; use 16 or 24 for IBM
p_test_run = .false.
l_test_openmp = .false.
l_log_checks = .false.
num_io_procs = 0 ! up to one PE per output stream is possible
itype_comm = 1
iorder_sendrecv = 3 ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
dynamics_grid_filename = 'iconR2B05_DOM01.nc'
dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
lredgrid_phys = .false.
lfeedback = .true.
ifeedback_type = 2
/
&initicon_nml
lconsistency_checks = .false.
init_mode = 2 ! operation mode 2: IFS
zpbl1 = 500.
zpbl2 = 1000.
! l_sst_in = .true.
/
&run_nml
num_lev = 90
lvert_nest = .true. ! use vertical nesting if a nest is active
! nsteps = ${nsteps} ! 50 ! 1200 ! 7200 !
! dtime = ${dtime} ! timestep in seconds
modelTimeStep = "${modelTimeStep}"
ldynamics = .TRUE. ! dynamics
ltransport = .true.
iforcing = 3 ! NWP forcing
ltestcase = .false. ! false: run with real data
msg_level = 7 ! print maximum wind speeds every 5 time steps
ltimer = .true. ! set .TRUE. for timer output
timers_level = 10 ! can be increased up to 10 for detailed timer output
output = "nml"
lart = ${lart}
/
&nwp_phy_nml
inwp_gscp = 1
inwp_convection = 1
inwp_radiation = 1
inwp_cldcover = 1
inwp_turb = 1
inwp_satad = 1
inwp_sso = 1
inwp_gwd = 1
inwp_surface = 1
icapdcycl = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
latm_above_top = .false., .true. ! the second entry refers to the nested domain (if present)
efdt_min_raylfric = 7200.
itype_z0 = 2
icpl_aero_conv = 1
icpl_aero_gscp = 1
! resolution-dependent settings - please choose the appropriate one
dt_rad = 2160.
dt_conv = 720.
dt_sso = 1440.
dt_gwd = 1440.
/
&nwp_tuning_nml
tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
itune_albedo = 1 ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
tkhmin = 0.75 ! new default since rev. 16527
tkmmin = 0.75 ! ""
pat_len = 100.
c_diff = 0.2
rat_sea = 8.5 ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
ltkesso = .true.
frcsmot = 0.2 ! these 2 switches together apply vertical smoothing of the TKE source terms
imode_frcsmot = 2 ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
itype_sher = 3
ltkeshs = .true.
a_hshr = 2.0
/
&lnd_nml
ntiles = 3 !!! 1 for assimilation cycle and forecast
nlev_snow = 3 !!! 1 for assimilation cycle and forecast
lmulti_snow = .true. !!! .false. for assimilation cycle and forecast
itype_heatcond = 2
idiag_snowfrac = 2
lsnowtile = .false. !! later on .true. if GRIB encoding issues are solved
lseaice = .true.
llake = .false.
itype_lndtbl = 3 ! minimizes moist/cold bias in lower tropical troposphere
itype_root = 2
/
&radiation_nml
irad_o3 = 10
irad_aero = 6
albedo_type = 2 ! Modis albedo
vmr_co2 = 390.e-06 ! values representative for 2012
vmr_ch4 = 1800.e-09
vmr_n2o = 322.0e-09
vmr_o2 = 0.20946
vmr_cfc11 = 240.e-12
vmr_cfc12 = 532.e-12
/
&nonhydrostatic_nml
iadv_rhotheta = 2
ivctype = 2
itime_scheme = 4
exner_expol = 0.333
vwind_offctr = 0.2
damp_height = 50000.
rayleigh_coeff = 0.10
lhdiff_rcf = .true.
divdamp_order = 24 ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves
divdamp_type = 32 !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
divdamp_fac = 0.004
l_open_ubc = .false.
igradp_method = 3
l_zdiffu_t = .true.
thslp_zdiffu = 0.02
thhgtd_zdiffu = 125.
htop_moist_proc= 22500.
hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
min_lay_thckn = 20.
max_lay_thckn = 400. ! maximum layer thickness below htop_thcknlimit
htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
top_height = 75000.
stretch_fac = 0.9
decay_scale_1 = 4000.
decay_scale_2 = 2500.
decay_exp = 1.2
flat_height = 16000.
/
&dynamics_nml
iequations = 3
idiv_method = 1
divavg_cntrwgt = 0.50
lcoriolis = .TRUE.
/
&transport_nml
! qv, qc, qi, qr, qs
itype_vlimit = 1,1,1,1,1
ivadv_tracer = 3, 3, 3, 3, 3
itype_hlimit = 3, 4, 4, 4 , 4
ihadv_tracer = 52, 2,2,2,2
iadv_tke = 0
/
&diffusion_nml
hdiff_order = 5
itype_vn_diffu = 1
itype_t_diffu = 2
hdiff_efdt_ratio = 24.0
hdiff_smag_fac = 0.025
lhdiff_vn = .TRUE.
lhdiff_temp = .TRUE.
/
&interpol_nml
nudge_zone_width = 8
lsq_high_ord = 3
l_intp_c2l = .true.
l_mono_c2l = .true.
support_baryctr_intp = .false.
/
&extpar_nml
itopo = 1
n_iter_smooth_topo = 1
heightdiff_threshold = 3000.
/
&io_nml
itype_pres_msl = 4 ! IFS method with bug fix for self-consistency between SLP and geopotential
itype_rh = 1 ! RH w.r.t. water
/
&output_nml
filetype = ${FILETYPE} ! output format: 2=GRIB2, 4=NETCDFv2
dom = -1 ! write all domains
output_start = "${output_start}"
output_end = "${output_end}"
output_interval = "${output_interval}"
steps_per_file = 1
include_last = .TRUE.
output_filename = 'icon-art-${EXPNAME}-chem' ! file name base
ml_varlist = 'temp','pres','group:ART_CHEMISTRY','u','v'
output_grid = .TRUE.
remap = 1
reg_lon_def = -180.,1,180.
reg_lat_def = -90.,1,90.
/
</pre>
</div>
Please note in the last namelist section "output_nml" that you can set all output variables that you need to postprocess your data later. All assigned variables here will be written in the output netCDF-files as well. To learn more about post processing your data, check out a later chapter of this article or the [[Postprocessing]] article.

Now, we're getting to the ICON-ART settings. To enable chemistry in an ICON-ART Simulation inn general, the switch <code>lart_chem</code> has to be set to <code>.TRUE.</code>. With <code>lart_diag_out</code> output of the diagnostic fields can be enabled. Due to setting <code>lart_chem=.TRUE.</code> either <code>lart_chemtracer</code> or <code>lart_mecca</code> have to be set to <code>.TRUE.</code>. Because we want to perform a simulation with simplified chemistry, we have to switch on <code>lart_chemtracer</code>. If this namelist parameter is set to <code>.TRUE.</code>, also <code>cart_chemtracer_xml</code> has to be fulfilled. Here you enter the path of your xml-file which describes the tracers occurring and their properties in the simulation. How to create this xml-file is explained in the next chapter.
An example configuration for this part is shown in the following:
<div class="toccolours mw-collapsible mw-collapsed">
Part 4: Runscript ICON-ART Settings (Example configuration)
<pre class="mw-collapsible-content">
&art_nml
lart_chem = .TRUE.
lart_diag_out = .TRUE.
lart_aerosol = .FALSE.
lart_mecca = .FALSE.
lart_chemtracer = .TRUE.
cart_chemtracer_xml = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
cart_input_folder = '${OUTDIR}'
cart_io_suffix = '0014'
iart_init_gas = 0
/
EOF
</pre>
</div>
Please note that there are also several other namelist parameter you can select from (see [[Namelist]] article) but to perform our case study we're done for the ART setting at this point.

Depending on the used HPC-System, some parameter concerning the running job like maximum running time and used nodes can be set. For this case study the following settings can be copied. Note that this is valid for the HoreKa HPC system and that it can differ to other systems.
<div class="toccolours mw-collapsible mw-collapsed">
Part 5: Runscript job Settings (Example configuration)
<pre class="mw-collapsible-content">
cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=2
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON
</pre>
</div>

To conclude and to double check, in the following box the complete runscript is shown once again.

<div class="toccolours mw-collapsible mw-collapsed">
Complete example configuration of the runscript
<pre class="mw-collapsible-content">
#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-liftime_tracer_test
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------
! run_nml: general switches ----------
&amp;# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-02T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-02T00:00:00Z"}
output_interval="PT1H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"

# model parameters
model_equations=3 # equation system
# 1=hydrost. atm. T
# 1=hydrost. atm. theta dp
# 3=non-hydrost. atm.,
# 0=shallow water model
# -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
lRestart = .false.
/
&master_time_control_nml
experimentStartDate = "$start_date"
experimentStopDate = "$end_date"
forecastLeadTime = "$leadtime"
checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
model_type=1
model_name="ATMO"
model_namelist_filename="$atmo_namelist"
model_min_rank=1
model_max_rank=65536
model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
nproma = 8 ! optimal setting 8 for CRAY; use 16 or 24 for IBM
p_test_run = .false.
l_test_openmp = .false.
l_log_checks = .false.
num_io_procs = 0 ! up to one PE per output stream is possible
itype_comm = 1
iorder_sendrecv = 3 ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
dynamics_grid_filename = 'iconR2B05_DOM01.nc'
dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
lredgrid_phys = .false.
lfeedback = .true.
ifeedback_type = 2
/
&initicon_nml
lconsistency_checks = .false.
init_mode = 2 ! operation mode 2: IFS
zpbl1 = 500.
zpbl2 = 1000.
! l_sst_in = .true.
/
&run_nml
num_lev = 90
lvert_nest = .true. ! use vertical nesting if a nest is active
! nsteps = ${nsteps} ! 50 ! 1200 ! 7200 !
! dtime = ${dtime} ! timestep in seconds
modelTimeStep = "${modelTimeStep}"
ldynamics = .TRUE. ! dynamics
ltransport = .true.
iforcing = 3 ! NWP forcing
ltestcase = .false. ! false: run with real data
msg_level = 7 ! print maximum wind speeds every 5 time steps
ltimer = .true. ! set .TRUE. for timer output
timers_level = 10 ! can be increased up to 10 for detailed timer output
output = "nml"
lart = ${lart}
/
&nwp_phy_nml
inwp_gscp = 1
inwp_convection = 1
inwp_radiation = 1
inwp_cldcover = 1
inwp_turb = 1
inwp_satad = 1
inwp_sso = 1
inwp_gwd = 1
inwp_surface = 1
icapdcycl = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
latm_above_top = .false., .true. ! the second entry refers to the nested domain (if present)
efdt_min_raylfric = 7200.
itype_z0 = 2
icpl_aero_conv = 1
icpl_aero_gscp = 1
! resolution-dependent settings - please choose the appropriate one
dt_rad = 2160.
dt_conv = 720.
dt_sso = 1440.
dt_gwd = 1440.
/
&nwp_tuning_nml
tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
itune_albedo = 1 ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
tkhmin = 0.75 ! new default since rev. 16527
tkmmin = 0.75 ! ""
pat_len = 100.
c_diff = 0.2
rat_sea = 8.5 ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
ltkesso = .true.
frcsmot = 0.2 ! these 2 switches together apply vertical smoothing of the TKE source terms
imode_frcsmot = 2 ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
itype_sher = 3
ltkeshs = .true.
a_hshr = 2.0
/
&lnd_nml
ntiles = 3 !!! 1 for assimilation cycle and forecast
nlev_snow = 3 !!! 1 for assimilation cycle and forecast
lmulti_snow = .true. !!! .false. for assimilation cycle and forecast
itype_heatcond = 2
idiag_snowfrac = 2
lsnowtile = .false. !! later on .true. if GRIB encoding issues are solved
lseaice = .true.
llake = .false.
itype_lndtbl = 3 ! minimizes moist/cold bias in lower tropical troposphere
itype_root = 2
/
&radiation_nml
irad_o3 = 10
irad_aero = 6
albedo_type = 2 ! Modis albedo
vmr_co2 = 390.e-06 ! values representative for 2012
vmr_ch4 = 1800.e-09
vmr_n2o = 322.0e-09
vmr_o2 = 0.20946
vmr_cfc11 = 240.e-12
vmr_cfc12 = 532.e-12
/
&nonhydrostatic_nml
iadv_rhotheta = 2
ivctype = 2
itime_scheme = 4
exner_expol = 0.333
vwind_offctr = 0.2
damp_height = 50000.
rayleigh_coeff = 0.10
lhdiff_rcf = .true.
divdamp_order = 24 ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves
divdamp_type = 32 !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
divdamp_fac = 0.004
l_open_ubc = .false.
igradp_method = 3
l_zdiffu_t = .true.
thslp_zdiffu = 0.02
thhgtd_zdiffu = 125.
htop_moist_proc= 22500.
hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
min_lay_thckn = 20.
max_lay_thckn = 400. ! maximum layer thickness below htop_thcknlimit
htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
top_height = 75000.
stretch_fac = 0.9
decay_scale_1 = 4000.
decay_scale_2 = 2500.
decay_exp = 1.2
flat_height = 16000.
/
&dynamics_nml
iequations = 3
idiv_method = 1
divavg_cntrwgt = 0.50
lcoriolis = .TRUE.
/
&transport_nml
! qv, qc, qi, qr, qs
itype_vlimit = 1,1,1,1,1
ivadv_tracer = 3, 3, 3, 3, 3
itype_hlimit = 3, 4, 4, 4 , 4
ihadv_tracer = 52, 2,2,2,2
iadv_tke = 0
/
&diffusion_nml
hdiff_order = 5
itype_vn_diffu = 1
itype_t_diffu = 2
hdiff_efdt_ratio = 24.0
hdiff_smag_fac = 0.025
lhdiff_vn = .TRUE.
lhdiff_temp = .TRUE.
/
&interpol_nml
nudge_zone_width = 8
lsq_high_ord = 3
l_intp_c2l = .true.
l_mono_c2l = .true.
support_baryctr_intp = .false.
/
&extpar_nml
itopo = 1
n_iter_smooth_topo = 1
heightdiff_threshold = 3000.
/
&io_nml
itype_pres_msl = 4 ! IFS method with bug fix for self-consistency between SLP and geopotential
itype_rh = 1 ! RH w.r.t. water
/
&output_nml
filetype = ${FILETYPE} ! output format: 2=GRIB2, 4=NETCDFv2
dom = -1 ! write all domains
output_start = "${output_start}"
output_end = "${output_end}"
output_interval = "${output_interval}"
steps_per_file = 1
include_last = .TRUE.
output_filename = 'icon-art-${EXPNAME}-chem' ! file name base
ml_varlist = 'temp','pres','group:ART_CHEMISTRY','u','v'
output_grid = .TRUE.
remap = 1
reg_lon_def = -180.,1,180.
reg_lat_def = -90.,1,90.
/

&art_nml
lart_chem = .TRUE.
lart_diag_out = .TRUE.
lart_aerosol = .FALSE.
lart_mecca = .FALSE.
lart_chemtracer = .TRUE.
cart_chemtracer_xml = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
cart_input_folder = '${OUTDIR}'
cart_io_suffix = '0014'
iart_init_gas = 0
/
EOF

cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=2
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON
</pre>
</div>

Revision as of 14:11, 20 July 2023

- under construction -

In this example it is shown how to simulate a lifetime driven tracer with simplified chemistry in ICON-ART. This tutorial teaches you...

  • the setup of the runscript
  • the correct setup of the xml-file for such tracers.
  • the structure of lifetime tracers in xml-files
  • editing output variables


Configuration case

The depicted case is dealing with biogenic very short-lived species (VSLS) with a very short lifetime, more accurately Bromoform () and Dibromomethane (). Like of most VSLSs the major source of and is the ocean which leads too a large gradient with increasing height in the concentration of these tracers. Bromoform is mainly depleted by photolysis in the Troposphere whereas Dibromomethanes main loss is due to Hydroxylradicals (OH). To assess the ability to simulate the transport of VSLS from the surface to the lower stratosphere, this case study uses an idealized chemical tracer approach.

The simulation is modeling the 01 October 2012 and is initialized with data from the ECMWF Integrated Forecast System (IFS) and includes boundary conditions and chemical lifetimes from the WMO Ozone assessment 2010. The boundary conditions and the chemical lifetimes are recalculated in a sort of rate at which the substances are depleted from the atmosphere with help of the implicit solution of the balance equation: Failed to parse (syntax error): {\displaystyle \frac{\partial(\bar{\rho)\hat{\Psi_{l}}/{\partial t} = -\nabla\cdot (\hat{v}\bar{\rho}\hat{\Psi_{l}})- \nabla\cdot\bar{(\rho v''\Psi_{g,l}''}+P_l-L_l+E_l}

Here is the density of air, the barycentric-averaged mass mixing ratio, indicates the flux divergence that includes the horizontal and vertical advection of the gaseous compound l and indicates the change due to turbulent fluxes. Further describes the production rate due to chemical reactions, the respective loss rate and emissions are noted with . Everything is related to the respective compound l.

In this example no emission data is used.


Setting up the Runscript

Let's start with the runscript that has to be prepared. Please note that the in the following explained parts have to be printed in one runscript-file with the naming designation "xyz.run". Here it is named lifetime_tracer_test.run but of course you can call it differently as well.

If you've also worked through the example of the Simplified Chemistry, you can use nearly the same runscript. Note that you have to change the paths from Part 1, the timing settings from Part 2, the output variables from Part 3, the emission settings as well as the path of the chemtracer-xml-file in the ART-settings from Part 4 and finally the timing in the job settings from Part 5. Details can be found below.

Inside of that, first check that all your directories are correct, probably they have to be adjusted. Abbreviations used here are the following:

  • CENTER: Your organization
  • EXPNAME: name of your ICON-Simulation
  • OUTDIR: Directory where the simulation output will be stored
  • ARTFOLDER: Directory where the ICON-ART code is stored
  • INDIR: Directory where the necessary Input data are stored
  • EXP:
  • lart: For ICON-ART Simulation that has to be switched to Oheim_simple_icon.run.

Part 1: Runscript Directory Settings (Example configuration) 

#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-liftime_tracer_test
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder  from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
    mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat      ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat           ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat      ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat      ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat      ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat          ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat         ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat      ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat          ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------

Additionally in the next lines of code you set the timing. In this simulation we only simulate one day (01 October 2012). To really catch all day times and so the time dependent solar radiation, the output interval is set to 1 hours to calculate and to every time of the day. Because of the Photolysis dependency of this is particularly important.

Part 2: Runscript Timing Settings (Example configuration) 

! run_nml: general switches ----------
&# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-02T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-02T00:00:00Z"}
output_interval="PT1H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"

Further, all the namelist parameters (from the regular ICON model without ART-extension) have to be set. For a regular ICON-ART-Simulation the following settings are recommended - if not stated differently. For a detailed description, check out the ICON Documentation (Drill et. al. (2019)).

Part 3: Runscript ICON-Parameter and -Namelist Settings (Example configuration) 

# model parameters
model_equations=3     # equation system
#                     1=hydrost. atm. T
#                     1=hydrost. atm. theta dp
#                     3=non-hydrost. atm.,
#                     0=shallow water model
#                    -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
 lRestart               = .false.
/
&master_time_control_nml
 experimentStartDate = "$start_date"
 experimentStopDate  = "$end_date"
 forecastLeadTime = "$leadtime"
 checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
  model_type=1
  model_name="ATMO"
  model_namelist_filename="$atmo_namelist"
  model_min_rank=1
  model_max_rank=65536
  model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
  dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
  radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
 nproma         = 8  ! optimal setting 8 for CRAY; use 16 or 24 for IBM
 p_test_run     = .false.
 l_test_openmp  = .false.
 l_log_checks   = .false.
 num_io_procs   = 0   ! up to one PE per output stream is possible
 itype_comm     = 1
 iorder_sendrecv = 3  ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
 dynamics_grid_filename  = 'iconR2B05_DOM01.nc'
 dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
 lredgrid_phys           = .false.
 lfeedback               = .true.
 ifeedback_type          = 2
/
&initicon_nml
 lconsistency_checks      = .false.
 init_mode   = 2           ! operation mode 2: IFS
 zpbl1       = 500. 
 zpbl2       = 1000. 
! l_sst_in    = .true.
/
&run_nml
 num_lev        = 90
 lvert_nest     = .true.       ! use vertical nesting if a nest is active
! nsteps         = ${nsteps}    ! 50 ! 1200 ! 7200 !
! dtime          = ${dtime}     ! timestep in seconds
 modelTimeStep  = "${modelTimeStep}"
 ldynamics      = .TRUE.       ! dynamics
 ltransport     = .true.
 iforcing       = 3            ! NWP forcing
 ltestcase      = .false.      ! false: run with real data
 msg_level      = 7            ! print maximum wind speeds every 5 time steps
 ltimer         = .true.      ! set .TRUE. for timer output
 timers_level   = 10            ! can be increased up to 10 for detailed timer output
 output         = "nml"
 lart           = ${lart}
/
&nwp_phy_nml
 inwp_gscp       = 1
 inwp_convection = 1
 inwp_radiation  = 1
 inwp_cldcover   = 1
 inwp_turb       = 1
 inwp_satad      = 1
 inwp_sso        = 1
 inwp_gwd        = 1
 inwp_surface    = 1
 icapdcycl       = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
 latm_above_top  = .false., .true.  ! the second entry refers to the nested domain (if present)
 efdt_min_raylfric = 7200.
 itype_z0         = 2
 icpl_aero_conv   = 1
 icpl_aero_gscp   = 1
 ! resolution-dependent settings - please choose the appropriate one
 dt_rad    = 2160.
 dt_conv   = 720.
 dt_sso    = 1440.
 dt_gwd    = 1440.
/
&nwp_tuning_nml
 tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
 itune_albedo   = 1     ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
 tkhmin  = 0.75  ! new default since rev. 16527
 tkmmin  = 0.75  !           "" 
 pat_len = 100.
 c_diff  = 0.2
 rat_sea = 8.5  ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
 ltkesso = .true.
 frcsmot = 0.2      ! these 2 switches together apply vertical smoothing of the TKE source terms
 imode_frcsmot = 2  ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
 ! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
 itype_sher = 3    
 ltkeshs    = .true.
 a_hshr     = 2.0
/
&lnd_nml
 ntiles         = 3      !!! 1 for assimilation cycle and forecast
 nlev_snow      = 3      !!! 1 for assimilation cycle and forecast
 lmulti_snow    = .true. !!! .false. for assimilation cycle and forecast
 itype_heatcond = 2
 idiag_snowfrac = 2
 lsnowtile      = .false.  !! later on .true. if GRIB encoding issues are solved
 lseaice        = .true.
 llake          = .false.
 itype_lndtbl   = 3  ! minimizes moist/cold bias in lower tropical troposphere
 itype_root     = 2
/
&radiation_nml
 irad_o3       = 10
 irad_aero     = 6
 albedo_type   = 2 ! Modis albedo
 vmr_co2       = 390.e-06 ! values representative for 2012
 vmr_ch4       = 1800.e-09
 vmr_n2o       = 322.0e-09
 vmr_o2        = 0.20946
 vmr_cfc11     = 240.e-12
 vmr_cfc12     = 532.e-12
/
&nonhydrostatic_nml
 iadv_rhotheta  = 2
 ivctype        = 2
 itime_scheme   = 4
 exner_expol    = 0.333
 vwind_offctr   = 0.2
 damp_height    = 50000.
 rayleigh_coeff = 0.10
 lhdiff_rcf     = .true.
 divdamp_order  = 24    ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves 
 divdamp_type   = 32    !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
 divdamp_fac    = 0.004
 l_open_ubc     = .false.
 igradp_method  = 3
 l_zdiffu_t     = .true.
 thslp_zdiffu   = 0.02
 thhgtd_zdiffu  = 125.
 htop_moist_proc= 22500.
 hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
 min_lay_thckn   = 20.
 max_lay_thckn   = 400.   ! maximum layer thickness below htop_thcknlimit
 htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
 top_height      = 75000.
 stretch_fac     = 0.9
 decay_scale_1   = 4000.
 decay_scale_2   = 2500.
 decay_exp       = 1.2
 flat_height     = 16000.
/
&dynamics_nml
 iequations     = 3
 idiv_method    = 1
 divavg_cntrwgt = 0.50
 lcoriolis      = .TRUE.
/
&transport_nml
!                qv, qc, qi, qr, qs
 itype_vlimit  = 1,1,1,1,1
 ivadv_tracer = 3, 3, 3, 3, 3
 itype_hlimit = 3, 4, 4, 4 , 4
 ihadv_tracer = 52, 2,2,2,2
 iadv_tke      = 0
/
&diffusion_nml
 hdiff_order      = 5
 itype_vn_diffu   = 1
 itype_t_diffu    = 2
 hdiff_efdt_ratio = 24.0
 hdiff_smag_fac   = 0.025
 lhdiff_vn        = .TRUE.
 lhdiff_temp      = .TRUE.
/
&interpol_nml
 nudge_zone_width  = 8
 lsq_high_ord      = 3
 l_intp_c2l        = .true.
 l_mono_c2l        = .true.
 support_baryctr_intp = .false.
/
&extpar_nml
 itopo          = 1
 n_iter_smooth_topo = 1
 heightdiff_threshold = 3000.
/
&io_nml
 itype_pres_msl = 4  ! IFS method with bug fix for self-consistency between SLP and geopotential
 itype_rh       = 1  ! RH w.r.t. water
/
&output_nml
 filetype                     =  ${FILETYPE}         ! output format: 2=GRIB2, 4=NETCDFv2
 dom                          =  -1                  ! write all domains
 output_start     = "${output_start}"
 output_end       = "${output_end}"
 output_interval  = "${output_interval}"
 steps_per_file               = 1
 include_last                 =  .TRUE.
 output_filename              = 'icon-art-${EXPNAME}-chem'                ! file name base
 ml_varlist                   = 'temp','pres','group:ART_CHEMISTRY','u','v'
 output_grid                  =  .TRUE.
 remap                        = 1
 reg_lon_def                  = -180.,1,180.
 reg_lat_def                  = -90.,1,90.
/

Please note in the last namelist section "output_nml" that you can set all output variables that you need to postprocess your data later. All assigned variables here will be written in the output netCDF-files as well. To learn more about post processing your data, check out a later chapter of this article or the Postprocessing article.

Now, we're getting to the ICON-ART settings. To enable chemistry in an ICON-ART Simulation inn general, the switch lart_chem has to be set to .TRUE.. With lart_diag_out output of the diagnostic fields can be enabled. Due to setting lart_chem=.TRUE. either lart_chemtracer or lart_mecca have to be set to .TRUE.. Because we want to perform a simulation with simplified chemistry, we have to switch on lart_chemtracer. If this namelist parameter is set to .TRUE., also cart_chemtracer_xml has to be fulfilled. Here you enter the path of your xml-file which describes the tracers occurring and their properties in the simulation. How to create this xml-file is explained in the next chapter. An example configuration for this part is shown in the following:

Part 4: Runscript ICON-ART Settings (Example configuration) 

&art_nml
 lart_chem       = .TRUE.
 lart_diag_out   = .TRUE.
 lart_aerosol    = .FALSE.
 lart_mecca      = .FALSE.
 lart_chemtracer = .TRUE.
 cart_chemtracer_xml   = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
 cart_input_folder     = '${OUTDIR}'
 cart_io_suffix        = '0014'
 iart_init_gas         =  0
/
EOF

Please note that there are also several other namelist parameter you can select from (see Namelist article) but to perform our case study we're done for the ART setting at this point.

Depending on the used HPC-System, some parameter concerning the running job like maximum running time and used nodes can be set. For this case study the following settings can be copied. Note that this is valid for the HoreKa HPC system and that it can differ to other systems.

Part 5: Runscript job Settings (Example configuration) 

cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=2
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON

To conclude and to double check, in the following box the complete runscript is shown once again.

Complete example configuration of the runscript 

#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-liftime_tracer_test
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder  from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
    mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat      ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat           ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat      ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat      ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat      ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat          ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat         ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat      ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat          ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------
! run_nml: general switches ----------
&# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-02T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-02T00:00:00Z"}
output_interval="PT1H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"

# model parameters
model_equations=3     # equation system
#                     1=hydrost. atm. T
#                     1=hydrost. atm. theta dp
#                     3=non-hydrost. atm.,
#                     0=shallow water model
#                    -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
 lRestart               = .false.
/
&master_time_control_nml
 experimentStartDate = "$start_date"
 experimentStopDate  = "$end_date"
 forecastLeadTime = "$leadtime"
 checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
  model_type=1
  model_name="ATMO"
  model_namelist_filename="$atmo_namelist"
  model_min_rank=1
  model_max_rank=65536
  model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
  dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
  radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
 nproma         = 8  ! optimal setting 8 for CRAY; use 16 or 24 for IBM
 p_test_run     = .false.
 l_test_openmp  = .false.
 l_log_checks   = .false.
 num_io_procs   = 0   ! up to one PE per output stream is possible
 itype_comm     = 1
 iorder_sendrecv = 3  ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
 dynamics_grid_filename  = 'iconR2B05_DOM01.nc'
 dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
 lredgrid_phys           = .false.
 lfeedback               = .true.
 ifeedback_type          = 2
/
&initicon_nml
 lconsistency_checks      = .false.
 init_mode   = 2           ! operation mode 2: IFS
 zpbl1       = 500. 
 zpbl2       = 1000. 
! l_sst_in    = .true.
/
&run_nml
 num_lev        = 90
 lvert_nest     = .true.       ! use vertical nesting if a nest is active
! nsteps         = ${nsteps}    ! 50 ! 1200 ! 7200 !
! dtime          = ${dtime}     ! timestep in seconds
 modelTimeStep  = "${modelTimeStep}"
 ldynamics      = .TRUE.       ! dynamics
 ltransport     = .true.
 iforcing       = 3            ! NWP forcing
 ltestcase      = .false.      ! false: run with real data
 msg_level      = 7            ! print maximum wind speeds every 5 time steps
 ltimer         = .true.      ! set .TRUE. for timer output
 timers_level   = 10            ! can be increased up to 10 for detailed timer output
 output         = "nml"
 lart           = ${lart}
/
&nwp_phy_nml
 inwp_gscp       = 1
 inwp_convection = 1
 inwp_radiation  = 1
 inwp_cldcover   = 1
 inwp_turb       = 1
 inwp_satad      = 1
 inwp_sso        = 1
 inwp_gwd        = 1
 inwp_surface    = 1
 icapdcycl       = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
 latm_above_top  = .false., .true.  ! the second entry refers to the nested domain (if present)
 efdt_min_raylfric = 7200.
 itype_z0         = 2
 icpl_aero_conv   = 1
 icpl_aero_gscp   = 1
 ! resolution-dependent settings - please choose the appropriate one
 dt_rad    = 2160.
 dt_conv   = 720.
 dt_sso    = 1440.
 dt_gwd    = 1440.
/
&nwp_tuning_nml
 tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
 itune_albedo   = 1     ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
 tkhmin  = 0.75  ! new default since rev. 16527
 tkmmin  = 0.75  !           "" 
 pat_len = 100.
 c_diff  = 0.2
 rat_sea = 8.5  ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
 ltkesso = .true.
 frcsmot = 0.2      ! these 2 switches together apply vertical smoothing of the TKE source terms
 imode_frcsmot = 2  ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
 ! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
 itype_sher = 3    
 ltkeshs    = .true.
 a_hshr     = 2.0
/
&lnd_nml
 ntiles         = 3      !!! 1 for assimilation cycle and forecast
 nlev_snow      = 3      !!! 1 for assimilation cycle and forecast
 lmulti_snow    = .true. !!! .false. for assimilation cycle and forecast
 itype_heatcond = 2
 idiag_snowfrac = 2
 lsnowtile      = .false.  !! later on .true. if GRIB encoding issues are solved
 lseaice        = .true.
 llake          = .false.
 itype_lndtbl   = 3  ! minimizes moist/cold bias in lower tropical troposphere
 itype_root     = 2
/
&radiation_nml
 irad_o3       = 10
 irad_aero     = 6
 albedo_type   = 2 ! Modis albedo
 vmr_co2       = 390.e-06 ! values representative for 2012
 vmr_ch4       = 1800.e-09
 vmr_n2o       = 322.0e-09
 vmr_o2        = 0.20946
 vmr_cfc11     = 240.e-12
 vmr_cfc12     = 532.e-12
/
&nonhydrostatic_nml
 iadv_rhotheta  = 2
 ivctype        = 2
 itime_scheme   = 4
 exner_expol    = 0.333
 vwind_offctr   = 0.2
 damp_height    = 50000.
 rayleigh_coeff = 0.10
 lhdiff_rcf     = .true.
 divdamp_order  = 24    ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves 
 divdamp_type   = 32    !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
 divdamp_fac    = 0.004
 l_open_ubc     = .false.
 igradp_method  = 3
 l_zdiffu_t     = .true.
 thslp_zdiffu   = 0.02
 thhgtd_zdiffu  = 125.
 htop_moist_proc= 22500.
 hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
 min_lay_thckn   = 20.
 max_lay_thckn   = 400.   ! maximum layer thickness below htop_thcknlimit
 htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
 top_height      = 75000.
 stretch_fac     = 0.9
 decay_scale_1   = 4000.
 decay_scale_2   = 2500.
 decay_exp       = 1.2
 flat_height     = 16000.
/
&dynamics_nml
 iequations     = 3
 idiv_method    = 1
 divavg_cntrwgt = 0.50
 lcoriolis      = .TRUE.
/
&transport_nml
!                qv, qc, qi, qr, qs
 itype_vlimit  = 1,1,1,1,1
 ivadv_tracer = 3, 3, 3, 3, 3
 itype_hlimit = 3, 4, 4, 4 , 4
 ihadv_tracer = 52, 2,2,2,2
 iadv_tke      = 0
/
&diffusion_nml
 hdiff_order      = 5
 itype_vn_diffu   = 1
 itype_t_diffu    = 2
 hdiff_efdt_ratio = 24.0
 hdiff_smag_fac   = 0.025
 lhdiff_vn        = .TRUE.
 lhdiff_temp      = .TRUE.
/
&interpol_nml
 nudge_zone_width  = 8
 lsq_high_ord      = 3
 l_intp_c2l        = .true.
 l_mono_c2l        = .true.
 support_baryctr_intp = .false.
/
&extpar_nml
 itopo          = 1
 n_iter_smooth_topo = 1
 heightdiff_threshold = 3000.
/
&io_nml
 itype_pres_msl = 4  ! IFS method with bug fix for self-consistency between SLP and geopotential
 itype_rh       = 1  ! RH w.r.t. water
/
&output_nml
 filetype                     =  ${FILETYPE}         ! output format: 2=GRIB2, 4=NETCDFv2
 dom                          =  -1                  ! write all domains
 output_start     = "${output_start}"
 output_end       = "${output_end}"
 output_interval  = "${output_interval}"
 steps_per_file               = 1
 include_last                 =  .TRUE.
 output_filename              = 'icon-art-${EXPNAME}-chem'                ! file name base
 ml_varlist                   = 'temp','pres','group:ART_CHEMISTRY','u','v'
 output_grid                  =  .TRUE.
 remap                        = 1
 reg_lon_def                  = -180.,1,180.
 reg_lat_def                  = -90.,1,90.
/

&art_nml
 lart_chem       = .TRUE.
 lart_diag_out   = .TRUE.
 lart_aerosol    = .FALSE.
 lart_mecca      = .FALSE.
 lart_chemtracer = .TRUE.
 cart_chemtracer_xml   = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
 cart_input_folder     = '${OUTDIR}'
 cart_io_suffix        = '0014'
 iart_init_gas         =  0
/
EOF

cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=2
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON
Retrieved from "https://www.icon-art.kit.edu/userguide/index.php?title=Lifetime_Tracer_Simulation&oldid=259"