Stratospheric Ozone Chemistry and Polar Stratospheric Clouds

- under construction! -

In this example a simulation with only stratospheric (simplified) ozone chemistry is performed, also polar stratospheric clouds are taken into account. The tutorial teaches you...

• the implementation of stratospheric specific chemistry (here on thee example of ozone)
• applying linearized ozone chemistry (LINOZ) in a simulation
• the implementation of polar stratospheric clouds (PSCs)

No emission data will be included in this simulation.

Configuration case

Stratospheric ozone with linearized ozone chemistry (LINOZ)

On the one hand the depicted case is dealing with simulating stratospheric ozone which is calculated with a linearized ozone chemistry (LINOZ). Here a linearized version of the differential equation for production or depletion like ${\frac {\mathrm {d} c_{i}}{\mathrm {d} t}}=P_{i}-{\frac {c_{i}}{\tau _{i}}}$ is used. After calculating the Taylor expansion of first order the computed ozone concentrations are then anomalies from temperature and ozone column climatologies. This LINOZ chemistry is applied in heights of 10km and higher. Below that height the lifetime of ozone is set to constant 28 days so a constant climatology is applied. Without including LINOZ at this point, a complete ozone chemistry must be included in the simulation what would result in higher computational effort.

Polar Stratospheric Clouds (PSCs)

Further, the simulation of polar stratospheric clouds (PSC), are also considered. Polar stratospheric clouds are forming under very cold conditions in the antarctic region during the south hemispheric winter. These cold conditions are reached due to the underlying orographic conditions in Antarctica and a resulting very strong and stable polar vortex. In May or June the temperature is normally low enough to form polar stratospheric clouds. Then the reaction

${\ce {ClNO3 + HCl -> Cl2 + HNO3}}$

is effektively producing high ${\ce {HNO3}}$ concentrations in the stratosphere resulting in a formation of ${\ce {HNO3*H20}}$ phases, identifying as polar stratospheric clouds. The educts are existing due to the regular ${\ce {ClOx}}$ family reactions (see e.g. Dameris et al., 2007). Later ion the year, when more light reaches Antarctica, the ${\ce {Cl2}}$ product can then be photolysed and finally, due to the ${\ce {ClOx}}$ cycle, Ozone can be depleted:

${\ce {Cl2 + hv -> 2Cl}}$

${\ce {Cl + O3 -> ClO + O2}}$

That's also why the ozone hole has formed: The more chlorofluorocarbons are emitted, the more ${\ce {ClNO3}}$ as well as ${\ce {HCl}}$ are produced and the more ozone can be depleted. In this simulation, PSCs are considered to calculate the stratospheric ozone concentration but not to simulate the clouds itself.

Setting up the Runscript

Let's start with the runscript that has to be prepared. Please note that the in the following explained parts have to be printed in one runscript-file with the naming designation "xyz.run". Here it is named exp.testsuite.strat_ozone_linoz_psc.run but of course you can call it differently as well. The runscript can be stored under the following path in your icon directory: /icon-kit/run

If you've already walked through the example of the Simplified Chemistry, you can use nearly the same runscript. Note that you have to change the paths from Part 1, the timing settings from Part 2, the output variables from Part 3, the emission settings as well as the path of the chemtracer-xml-file in the ART-settings from Part 4 and finally the timing in the job settings from Part 5. Details can be found below.

Inside of that, first check in part 1 that all your paths to your directories are correct, probably they have to be adjusted. Note that "hp8526" is a name of a specific account here, make sure to double check especially these lines. Abbreviations used here are the following:

• CENTER: Your organization
• EXPNAME: name of your ICON-Simulation
• OUTDIR: Directory where the simulation output will be stored
• ARTFOLDER: Directory where the ICON-ART code is stored
• INDIR: Directory where the necessary Input data are stored
• EXP:
• lart: For ICON-ART Simulation that has to be switched to .True..

Expand

Part 1: Runscript Directory Settings (Example configuration)

#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-strat_ozone_linoz_psc
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder  from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
    mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat      ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat           ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat      ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat      ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat      ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat          ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat         ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat      ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat          ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------

Additionally in the next lines of code you set the timing. In this simulation we simulate one week (01 October 2012 until 07 October 2012). To really catch all day times and so the time dependent solar radiation, the output interval is set to 10 hours to calculate ${\ce {O3}}$ to every time of the day. Because of the Photolysis dependency of ${\ce {O3}}$ this is particularly important.

Expand

Part 2: Runscript Timing Settings (Example configuration)

! run_nml: general switches ----------
&# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-08T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-08T00:00:00Z"}
output_interval="PT10H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"

Further, all the namelist parameters (from the regular ICON model without ART-extension) have to be set. For a regular ICON-ART-Simulation the following settings are recommended - if not stated differently. For a detailed description, check out the ICON Documentation (Drill et. al. (2019)).

Since we make use of LINOZ, in the radiation namelist section &radiation_nml the namelist parameter irad_o3 has to be set to 10:

Expand

Part 3: Runscript ICON-Parameter and -Namelist Settings (Example configuration)

# model parameters
model_equations=3     # equation system
#                     1=hydrost. atm. T
#                     1=hydrost. atm. theta dp
#                     3=non-hydrost. atm.,
#                     0=shallow water model
#                    -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
 lRestart               = .false.
/
&master_time_control_nml
 experimentStartDate = "$start_date"
 experimentStopDate  = "$end_date"
 forecastLeadTime = "$leadtime"
 checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
  model_type=1
  model_name="ATMO"
  model_namelist_filename="$atmo_namelist"
  model_min_rank=1
  model_max_rank=65536
  model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
  dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
  radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
 nproma         = 8  ! optimal setting 8 for CRAY; use 16 or 24 for IBM
 p_test_run     = .false.
 l_test_openmp  = .false.
 l_log_checks   = .false.
 num_io_procs   = 0   ! up to one PE per output stream is possible
 itype_comm     = 1
 iorder_sendrecv = 3  ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
 dynamics_grid_filename  = 'iconR2B05_DOM01.nc'
 dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
 lredgrid_phys           = .false.
 lfeedback               = .true.
 ifeedback_type          = 2
/
&initicon_nml
 lconsistency_checks      = .false.
 init_mode   = 2           ! operation mode 2: IFS
 zpbl1       = 500. 
 zpbl2       = 1000. 
! l_sst_in    = .true.
/
&run_nml
 num_lev        = 90
 lvert_nest     = .true.       ! use vertical nesting if a nest is active
! nsteps         = ${nsteps}    ! 50 ! 1200 ! 7200 !
! dtime          = ${dtime}     ! timestep in seconds
 modelTimeStep  = "${modelTimeStep}"
 ldynamics      = .TRUE.       ! dynamics
 ltransport     = .true.
 iforcing       = 3            ! NWP forcing
 ltestcase      = .false.      ! false: run with real data
 msg_level      = 7            ! print maximum wind speeds every 5 time steps
 ltimer         = .true.      ! set .TRUE. for timer output
 timers_level   = 10            ! can be increased up to 10 for detailed timer output
 output         = "nml"
 lart           = ${lart}
/
&nwp_phy_nml
 inwp_gscp       = 1
 inwp_convection = 1
 inwp_radiation  = 1
 inwp_cldcover   = 1
 inwp_turb       = 1
 inwp_satad      = 1
 inwp_sso        = 1
 inwp_gwd        = 1
 inwp_surface    = 1
 icapdcycl       = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
 latm_above_top  = .false., .true.  ! the second entry refers to the nested domain (if present)
 efdt_min_raylfric = 7200.
 itype_z0         = 2
 icpl_aero_conv   = 1
 icpl_aero_gscp   = 1
 ! resolution-dependent settings - please choose the appropriate one
 dt_rad    = 2160.
 dt_conv   = 720.
 dt_sso    = 1440.
 dt_gwd    = 1440.
/
&nwp_tuning_nml
 tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
 itune_albedo   = 1     ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
 tkhmin  = 0.75  ! new default since rev. 16527
 tkmmin  = 0.75  !           "" 
 pat_len = 100.
 c_diff  = 0.2
 rat_sea = 8.5  ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
 ltkesso = .true.
 frcsmot = 0.2      ! these 2 switches together apply vertical smoothing of the TKE source terms
 imode_frcsmot = 2  ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
 ! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
 itype_sher = 3    
 ltkeshs    = .true.
 a_hshr     = 2.0
/
&lnd_nml
 ntiles         = 3      !!! 1 for assimilation cycle and forecast
 nlev_snow      = 3      !!! 1 for assimilation cycle and forecast
 lmulti_snow    = .true. !!! .false. for assimilation cycle and forecast
 itype_heatcond = 2
 idiag_snowfrac = 2
 lsnowtile      = .false.  !! later on .true. if GRIB encoding issues are solved
 lseaice        = .true.
 llake          = .false.
 itype_lndtbl   = 3  ! minimizes moist/cold bias in lower tropical troposphere
 itype_root     = 2
/
&radiation_nml
 irad_o3       = 10
 irad_aero     = 6
 albedo_type   = 2 ! Modis albedo
 vmr_co2       = 390.e-06 ! values representative for 2012
 vmr_ch4       = 1800.e-09
 vmr_n2o       = 322.0e-09
 vmr_o2        = 0.20946
 vmr_cfc11     = 240.e-12
 vmr_cfc12     = 532.e-12
/
&nonhydrostatic_nml
 iadv_rhotheta  = 2
 ivctype        = 2
 itime_scheme   = 4
 exner_expol    = 0.333
 vwind_offctr   = 0.2
 damp_height    = 50000.
 rayleigh_coeff = 0.10
 lhdiff_rcf     = .true.
 divdamp_order  = 24    ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves 
 divdamp_type   = 32    !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
 divdamp_fac    = 0.004
 l_open_ubc     = .false.
 igradp_method  = 3
 l_zdiffu_t     = .true.
 thslp_zdiffu   = 0.02
 thhgtd_zdiffu  = 125.
 htop_moist_proc= 22500.
 hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
 min_lay_thckn   = 20.
 max_lay_thckn   = 400.   ! maximum layer thickness below htop_thcknlimit
 htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
 top_height      = 75000.
 stretch_fac     = 0.9
 decay_scale_1   = 4000.
 decay_scale_2   = 2500.
 decay_exp       = 1.2
 flat_height     = 16000.
/
&dynamics_nml
 iequations     = 3
 idiv_method    = 1
 divavg_cntrwgt = 0.50
 lcoriolis      = .TRUE.
/
&transport_nml
!                qv, qc, qi, qr, qs
 itype_vlimit  = 1,1,1,1,1
 ivadv_tracer = 3, 3, 3, 3, 3
 itype_hlimit = 3, 4, 4, 4 , 4
 ihadv_tracer = 52, 2,2,2,2
 iadv_tke      = 0
/
&diffusion_nml
 hdiff_order      = 5
 itype_vn_diffu   = 1
 itype_t_diffu    = 2
 hdiff_efdt_ratio = 24.0
 hdiff_smag_fac   = 0.025
 lhdiff_vn        = .TRUE.
 lhdiff_temp      = .TRUE.
/
&interpol_nml
 nudge_zone_width  = 8
 lsq_high_ord      = 3
 l_intp_c2l        = .true.
 l_mono_c2l        = .true.
 support_baryctr_intp = .false.
/
&extpar_nml
 itopo          = 1
 n_iter_smooth_topo = 1
 heightdiff_threshold = 3000.
/
&io_nml
 itype_pres_msl = 4  ! IFS method with bug fix for self-consistency between SLP and geopotential
 itype_rh       = 1  ! RH w.r.t. water
/
&output_nml
 filetype                     =  ${FILETYPE}         ! output format: 2=GRIB2, 4=NETCDFv2
 dom                          =  -1                  ! write all domains
 output_start     = "${output_start}"
 output_end       = "${output_end}"
 output_interval  = "${output_interval}"
 steps_per_file               = 1
 include_last                 =  .TRUE.
 output_filename              = 'icon-art-${EXPNAME}-chem'                ! file name base
 ml_varlist                   = 'temp','pres','group:ART_CHEMISTRY','u','v'
 output_grid                  =  .TRUE.
 remap                        = 1
 reg_lon_def                  = -180.,1,180.
 reg_lat_def                  = -90.,1,90.
/

Please note in the last namelist section "output_nml" that you can set all output variables that you need to postprocess your data later. All assigned variables here will be written in the output netCDF-files as well. To learn more about post processing your data, check out a later chapter of this article or the Postprocessing article.

Now, we're getting to the ICON-ART settings. To enable chemistry in an ICON-ART Simulation inn general, the switch lart_chem has to be set to .TRUE.. With lart_diag_out output of the diagnostic fields can be enabled. Due to setting lart_chem=.TRUE. either lart_chemtracer or lart_mecca has to be set to .TRUE.. Because we want to perform a simulation with simplified chemistry, we have to switch on lart_chemtracer. If this namelist parameter is set to .TRUE., also cart_chemtracer_xml has to be fulfilled. Here you enter the path of your xml-file which describes the tracers occurring and their properties in the simulation, which is only ${\ce {O3}}$ in this case. How to create this xml-file is explained in the next chapter. Since PSCs shall be considered, the lart_psc switch has to be set to .TRUE.. Please also note the extra information that have to be given for ${\ce {O3}}$ in the xml-file that PSCs are considered correctly (see next chapter). An example configuration for this part is shown in the following:

Expand

Part 4: Runscript ICON-ART Settings (Example configuration)

&art_nml
 lart_chem       = .TRUE.
 lart_diag_out   = .TRUE.
 lart_aerosol    = .FALSE.
 lart_mecca      = .FALSE.
 lart_chemtracer = .TRUE.
 lart_psc        = .TRUE.
 cart_chemtracer_xml   = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
 cart_input_folder     = '${OUTDIR}'
 cart_io_suffix        = '0014'
 iart_init_gas         =  0
/
EOF

Depending on the used HPC-System, some parameter concerning the running job like maximum running time and used nodes can be set. For this case study the following settings can be copied. Note that this is valid for the HoreKa HPC system and that it can differ to other systems.

Expand

Part 5: Runscript job Settings (Example configuration)

cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=4
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON

To conclude and to double check, in the following box the complete runscript is shown once again.

Expand

Complete example configuration of the runscript

#!/bin/bash
CENTER=IMK
workspace=/hkfs/work/workspace/scratch/hp8526-strat_ozone_linoz_psc
basedir=${workspace}/icon-kit-testsuite
icon_data_poolFolder=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT/AMIP/amip_input
EXPNAME=liftime_tracer_test
OUTDIR=${workspace}/output/${EXPNAME}
ICONFOLDER=/home/hk-project-iconart/hp8526/icon-kit
ARTFOLDER=${ICONFOLDER}/externals/art
INDIR=/hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/INPUT
EXP=ECHAM_AMIP_LIFETIME
lart=.True.

FILETYPE=4
COMPILER=intel
restart=.False.
read_restart_namelists=.False.

# Remove folder  from OUTDIR for postprocessing output
OUTDIR_PREFIX=${workspace}

# Create output directory and go to this directory

if [ ! -d $OUTDIR ]; then
    mkdir -p $OUTDIR
fi

cd $OUTDIR

#input for global domain
ln -sf ${INDIR}/../INPUT/GRID/icon_grid_0014_R02B05_G.nc iconR2B05_DOM01.nc
ln -sf ${INDIR}/../INPUT/EXTPAR/icon_extpar_0014_R02B05_G.nc extpar_iconR2B05_DOM01.nc
ln -sf /hkfs/work/workspace/scratch/fb4738-dwd_ozone/icon/output/0014_R02B05/uc1_ifs_t1279_grb2_remap_rev832_0014_R02B05_2018010100.nc ifs2icon_R2B05_DOM01.nc

ln -sf $ICONFOLDER/data/rrtmg_lw.nc rrtmg_lw.nc
ln -sf $ICONFOLDER/data/ECHAM6_CldOptProps.nc ECHAM6_CldOptProps.nc

ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/mozart_coord.nc ${OUTDIR}/mozart_coord.nc
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Linoz2004Br.dat ${OUTDIR}/Linoz2004Br.dat
ln -sf ${ARTFOLDER}/runctrl_examples/init_ctrl/Simnoy2002.dat ${OUTDIR}/Simnoy2002.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-aer.dat      ${OUTDIR}/FJX_scat-aer.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j.dat           ${OUTDIR}/FJX_j2j.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-cld.dat      ${OUTDIR}/FJX_scat-cld.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-ssa.dat      ${OUTDIR}/FJX_scat-ssa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_scat-UMa.dat      ${OUTDIR}/FJX_scat-UMa.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec.dat          ${OUTDIR}/FJX_spec.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_std.dat         ${OUTDIR}/atmos_std.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/atmos_h2och4.dat      ${OUTDIR}/atmos_h2och4.dat

ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_j2j_extended.dat          ${OUTDIR}/FJX_j2j_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended.dat ${OUTDIR}/FJX_spec_extended.dat
ln -sf ${ARTFOLDER}/runctrl_examples/photo_ctrl/FJX_spec_extended_lyman.dat ${OUTDIR}/FJX_spec_extended_lyman.dat

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# this if condition is necessary because otherwise
# a new link in ${INDIR}/${EXP}/emiss_minimal is generated
# linking to itself
if [ ! -L ${OUTDIR}/emissions ]; then
ln -sd /home/hk-project-iconart/hp8526/emissions       ${OUTDIR}/emissions
fi

# the namelist filename
atmo_namelist=NAMELIST_${EXPNAME}
#
#-----------------------------------------------------------------------------
! run_nml: general switches ----------
&# model timing

start_date=${start_date:="2012-10-01T00:00:00Z"}
end_date=${end_date:="2012-10-08T00:00:00Z"}
output_start=${start_date:="2012-10-01T00:00:00Z"}
output_end=${end_date:="2012-10-08T00:00:00Z"}
output_interval="PT10H"
modelTimeStep="PT6M"
leadtime="P8H"
checkpoint_interval="P30D"

# model parameters
model_equations=3     # equation system
#                     1=hydrost. atm. T
#                     1=hydrost. atm. theta dp
#                     3=non-hydrost. atm.,
#                     0=shallow water model
#                    -1=hydrost. ocean
#-----------------------------------------------------------------------------
# the grid parameters
declare -a atmo_dyn_grids=("iconR2B04_DOM01.nc" "iconR2B05_DOM02.nc" "iconR2B06_DOM03.nc")
# "iconR2B08_DOM03.nc"
#atmo_dyn_grids="iconR2B07_DOM02.nc","iconR2B08_DOM03.nc"
#atmo_rad_grids="iconR2B06_DOM01.nc"
#-----------------------------------------------------------------------------

# create ICON master namelist
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf

no_of_models=1

cat > icon_master.namelist << EOF
&master_nml
 lRestart               = .false.
/
&master_time_control_nml
 experimentStartDate = "$start_date"
 experimentStopDate  = "$end_date"
 forecastLeadTime = "$leadtime"
 checkpointTimeIntval = "$checkpoint_interval"
/
&master_model_nml
  model_type=1
  model_name="ATMO"
  model_namelist_filename="$atmo_namelist"
  model_min_rank=1
  model_max_rank=65536
  model_inc_rank=1
/
EOF

#-----------------------------------------------------------------------------
#

#-----------------------------------------------------------------------------
#
# write ICON namelist parameters
# ------------------------
# For a complete list see Namelist_overview and Namelist_overview.pdf
#
# ------------------------
# reconstrcuct the grid parameters in namelist form
dynamics_grid_filename=""
for gridfile in ${atmo_dyn_grids}; do
  dynamics_grid_filename="${dynamics_grid_filename} '${gridfile}',"
done
radiation_grid_filename=""
for gridfile in ${atmo_rad_grids}; do
  radiation_grid_filename="${radiation_grid_filename} '${gridfile}',"
done



# ------------------------

cat > ${atmo_namelist} << EOF
&parallel_nml
 nproma         = 8  ! optimal setting 8 for CRAY; use 16 or 24 for IBM
 p_test_run     = .false.
 l_test_openmp  = .false.
 l_log_checks   = .false.
 num_io_procs   = 0   ! up to one PE per output stream is possible
 itype_comm     = 1
 iorder_sendrecv = 3  ! best value for CRAY (slightly faster than option 1)
/
&grid_nml
 dynamics_grid_filename  = 'iconR2B05_DOM01.nc'
 dynamics_parent_grid_id = 0
!radiation_grid_filename = ${radiation_grid_filename}
 lredgrid_phys           = .false.
 lfeedback               = .true.
 ifeedback_type          = 2
/
&initicon_nml
 lconsistency_checks      = .false.
 init_mode   = 2           ! operation mode 2: IFS
 zpbl1       = 500. 
 zpbl2       = 1000. 
! l_sst_in    = .true.
/
&run_nml
 num_lev        = 90
 lvert_nest     = .true.       ! use vertical nesting if a nest is active
! nsteps         = ${nsteps}    ! 50 ! 1200 ! 7200 !
! dtime          = ${dtime}     ! timestep in seconds
 modelTimeStep  = "${modelTimeStep}"
 ldynamics      = .TRUE.       ! dynamics
 ltransport     = .true.
 iforcing       = 3            ! NWP forcing
 ltestcase      = .false.      ! false: run with real data
 msg_level      = 7            ! print maximum wind speeds every 5 time steps
 ltimer         = .true.      ! set .TRUE. for timer output
 timers_level   = 10            ! can be increased up to 10 for detailed timer output
 output         = "nml"
 lart           = ${lart}
/
&nwp_phy_nml
 inwp_gscp       = 1
 inwp_convection = 1
 inwp_radiation  = 1
 inwp_cldcover   = 1
 inwp_turb       = 1
 inwp_satad      = 1
 inwp_sso        = 1
 inwp_gwd        = 1
 inwp_surface    = 1
 icapdcycl       = 3 ! apply CAPE modification to improve diurnalcycle over tropical land (optimizes NWP scores)
 latm_above_top  = .false., .true.  ! the second entry refers to the nested domain (if present)
 efdt_min_raylfric = 7200.
 itype_z0         = 2
 icpl_aero_conv   = 1
 icpl_aero_gscp   = 1
 ! resolution-dependent settings - please choose the appropriate one
 dt_rad    = 2160.
 dt_conv   = 720.
 dt_sso    = 1440.
 dt_gwd    = 1440.
/
&nwp_tuning_nml
 tune_zceff_min = 0.075 ! ** default value to be used for R3B7; use 0.05 for R2B6 in order to get similar temperature biases in upper troposphere **
 itune_albedo   = 1     ! somewhat reduced albedo (w.r.t. MODIS data) over Sahara in order to reduce cold bias
/
&turbdiff_nml
 tkhmin  = 0.75  ! new default since rev. 16527
 tkmmin  = 0.75  !           "" 
 pat_len = 100.
 c_diff  = 0.2
 rat_sea = 8.5  ! ** new since r20191: 8.5 for R3B7, 8.0 for R2B6 in order to get similar temperature biases in the tropics **
 ltkesso = .true.
 frcsmot = 0.2      ! these 2 switches together apply vertical smoothing of the TKE source terms
 imode_frcsmot = 2  ! in the tropics (only), which reduces the moist bias in the tropical lower troposphere
 ! use horizontal shear production terms with 1/SQRT(Ri) scaling to prevent unwanted side effects:
 itype_sher = 3    
 ltkeshs    = .true.
 a_hshr     = 2.0
/
&lnd_nml
 ntiles         = 3      !!! 1 for assimilation cycle and forecast
 nlev_snow      = 3      !!! 1 for assimilation cycle and forecast
 lmulti_snow    = .true. !!! .false. for assimilation cycle and forecast
 itype_heatcond = 2
 idiag_snowfrac = 2
 lsnowtile      = .false.  !! later on .true. if GRIB encoding issues are solved
 lseaice        = .true.
 llake          = .false.
 itype_lndtbl   = 3  ! minimizes moist/cold bias in lower tropical troposphere
 itype_root     = 2
/
&radiation_nml
 irad_o3       = 10
 irad_aero     = 6
 albedo_type   = 2 ! Modis albedo
 vmr_co2       = 390.e-06 ! values representative for 2012
 vmr_ch4       = 1800.e-09
 vmr_n2o       = 322.0e-09
 vmr_o2        = 0.20946
 vmr_cfc11     = 240.e-12
 vmr_cfc12     = 532.e-12
/
&nonhydrostatic_nml
 iadv_rhotheta  = 2
 ivctype        = 2
 itime_scheme   = 4
 exner_expol    = 0.333
 vwind_offctr   = 0.2
 damp_height    = 50000.
 rayleigh_coeff = 0.10
 lhdiff_rcf     = .true.
 divdamp_order  = 24    ! for data assimilation runs, '2' provides extra-strong filtering of gravity waves 
 divdamp_type   = 32    !!! optional: 2 for assimilation cycle if very strong gravity-wave filtering is desired
 divdamp_fac    = 0.004
 l_open_ubc     = .false.
 igradp_method  = 3
 l_zdiffu_t     = .true.
 thslp_zdiffu   = 0.02
 thhgtd_zdiffu  = 125.
 htop_moist_proc= 22500.
 hbot_qvsubstep = 22500. ! use 19000. with R3B7
/
&sleve_nml
 min_lay_thckn   = 20.
 max_lay_thckn   = 400.   ! maximum layer thickness below htop_thcknlimit
 htop_thcknlimit = 14000. ! this implies that the upcoming COSMO-EU nest will have 60 levels
 top_height      = 75000.
 stretch_fac     = 0.9
 decay_scale_1   = 4000.
 decay_scale_2   = 2500.
 decay_exp       = 1.2
 flat_height     = 16000.
/
&dynamics_nml
 iequations     = 3
 idiv_method    = 1
 divavg_cntrwgt = 0.50
 lcoriolis      = .TRUE.
/
&transport_nml
!                qv, qc, qi, qr, qs
 itype_vlimit  = 1,1,1,1,1
 ivadv_tracer = 3, 3, 3, 3, 3
 itype_hlimit = 3, 4, 4, 4 , 4
 ihadv_tracer = 52, 2,2,2,2
 iadv_tke      = 0
/
&diffusion_nml
 hdiff_order      = 5
 itype_vn_diffu   = 1
 itype_t_diffu    = 2
 hdiff_efdt_ratio = 24.0
 hdiff_smag_fac   = 0.025
 lhdiff_vn        = .TRUE.
 lhdiff_temp      = .TRUE.
/
&interpol_nml
 nudge_zone_width  = 8
 lsq_high_ord      = 3
 l_intp_c2l        = .true.
 l_mono_c2l        = .true.
 support_baryctr_intp = .false.
/
&extpar_nml
 itopo          = 1
 n_iter_smooth_topo = 1
 heightdiff_threshold = 3000.
/
&io_nml
 itype_pres_msl = 4  ! IFS method with bug fix for self-consistency between SLP and geopotential
 itype_rh       = 1  ! RH w.r.t. water
/
&output_nml
 filetype                     =  ${FILETYPE}         ! output format: 2=GRIB2, 4=NETCDFv2
 dom                          =  -1                  ! write all domains
 output_start     = "${output_start}"
 output_end       = "${output_end}"
 output_interval  = "${output_interval}"
 steps_per_file               = 1
 include_last                 =  .TRUE.
 output_filename              = 'icon-art-${EXPNAME}-chem'                ! file name base
 ml_varlist                   = 'temp','pres','group:ART_CHEMISTRY','u','v'
 output_grid                  =  .TRUE.
 remap                        = 1
 reg_lon_def                  = -180.,1,180.
 reg_lat_def                  = -90.,1,90.
/

&art_nml
 lart_chem       = .TRUE.
 lart_diag_out   = .TRUE.
 lart_aerosol    = .FALSE.
 lart_mecca      = .FALSE.
 lart_chemtracer = .TRUE.
 lart_psc        = .TRUE.
 cart_chemtracer_xml   = '/home/hk-project-iconart/hp8526/icon-kit/externals/art/runctrl_examples/xml_ctrl/chemtracer_lifetime_test.xml'
 cart_input_folder     = '${OUTDIR}'
 cart_io_suffix        = '0014'
 iart_init_gas         =  0
/
EOF

cp ${ICONFOLDER}/bin/icon ./icon.exe

cat > job_ICON << ENDFILE
#!/bin/bash -x
#SBATCH --nodes=4
#SBATCH --time=06:00:00
#SBATCH --ntasks-per-node=76
#SBATCH --partition=cpuonly
#SBATCH -A hk-project-iconart
###SBATCH --constraint=LSDF

module load compiler/intel/2022.0.2 mpi/openmpi/4.0 lib/netcdf/4.9_serial lib/hdf5/1.12_serial lib/netcdf-fortran/4.5_serial lib/eccodes/2.25.0 numlib/mkl/2022.0.2

mpirun --bind-to core --map-by core --report-bindings ./icon.exe

ENDFILE

chmod +x job_ICON
sbatch job_ICON