5. Rendering the templates

5.1. What is rendering?

Rendering the templates consists in creating the parameters file and job script for our QOCT-GRAD calculations, based on their Jinja templates. The Jinja language offers a relatively easy and intuitive way of creating those templates and allows for more flexibility and adaptability in CONTROL LAUNCHER.

The rendering process makes use of three main elements:

  • The Jinja templates, defining how the information is presented in the parameters file and the job script

  • The content needed to “fill” those templates with the information specific to each calculation, partially provided by a new YAML file, called the configuration file

  • The rendering function that will make the link between those two, i.e. filling the templates with the content

Those elements are presented in more details in the following subsections. If you haven’t consulted it yet, it is strongly suggested to take a look at the What you need to know section, which contains tutorials that might prove helpful in your understanding of this section.

5.2. Jinja templates

Rather than trying to explain how to create your templates in a vacuum, let’s consider two basic examples to illustrate this process.

Warning

All the Jinja templates must be placed inside the templates directory of CONTROL LAUNCHER.

5.2.1. Parameters file template

Todo

COMING SOON

5.2.2. Job script template

Todo

COMING SOON

5.3. YAML configuration file

Todo

COMING SOON

5.4. Rendering functions

The Jinja templates define how the information is presented in the parameters file and the job script. The configuration file, among others, defines the content specific to each calculation. To link the two, CONTROL LAUNCHER calls a function defined in the control_renderer.py file, called a rendering function, that creates the parameters file and the job script by filling the templates with their specific content.

Important

The rendering function will be called for each transition-configuration combination, i.e. for each transition defined by the transition function coupled with each configuration file.

5.4.1. General definition

All the rendering functions must be defined in the control_renderer.py file and need to obey some restrictions in order to be callable by CONTROL LAUNCHER:

  • They take six dictionaries as arguments: clusters_cfg, config, system, data, job_specs and misc (see the next subsection for details).

  • They must return two variables :

    • A dictionary where each key is the name of the file and each associated value the rendered content of that file, for example {<parameters_filename>:<parameters_content> ; <job_script_name>:<job_script_content>}.

    • The name of the rendered job script, needed by the main script to launch the job on the cluster.

If a problem arises when rendering the templates, a ControlError exception should be raised with a proper error message (see how to handle errors for more details).

5.4.2. The six arguments

As said in the previous subsection, the rendering functions take six dictionaries as arguments. Since those functions might want various information depending on each specific case, we tried to include as many pertinent details as you might want to refer to during your rendering process. Thus, the six dictionaries are defined as follows:

  • clusters_cfg is the content of the clusters configuration file.

  • config is the content of the YAML configuration file.

  • system is the variable built by the system modelling process.

  • data contains the paths towards the data directory and its content, created during the system modelling process and when determining the transitions:

    • main_path is the absolute path towards the the data directory, created during the system modelling process.

    • mime_path, momdip_mtx_path, eigenvalues_path, eigenvectors_path, transpose_path and momdip_es_mtx_path are the path towards each of the file created during the system modelling process.

    • init_path and target_path are the path towards the initial and target states file of the considered transition (defined by the transition function and placed inside the data directory).

  • job_specs contains the information about the resources requirements, defined by the job scaling process, as well as other details about the job:

    • profile is the name of the profile as it appears in the clusters configuration file and as it was given in the command line.

    • scale_index is the computed value of the scale_index.

    • cluster_name is the name of the cluster on which CONTROL LAUNCHER is running, as it was given in the command line.

    • scale_label, scale_limit, partition, walltime and memory are all the information associated to the chosen job scale.

  • misc contains all other pertinent details:

    • code_dir is the path towards the CONTROL LAUNCHER directory.

    • templates_dir is the path towards the templates directory in the CONTROL LAUNCHER directory.

    • source_name is the name of the source file (minus a possible extension).

    • source_content is the content of the source file (in the form of a list where each element is a line of the file).

    • mol_dir is the path towards the molecule directory, bearing the name of the source file and containing the data directory and the job subdirectories.

    • config_name is the name of the configuration file.

    • transition_label is the label of the considered transition.

    • transitions_list, the list of dictionaries built when determining the transitions.

5.4.3. Using the Jinja templates

In order to use and render the Jinja templates, the rendering functions call another function, named jinja_render and defined in the control_renderer.py file. This function is the one that makes the link between the templates and their specific content:

control_renderer.jinja_render(templates_dir: str, template_file: str, render_vars: dict)[source]

Renders a file based on its Jinja template.

Parameters:

  • templates_dir (str) – The path towards the directory where the Jinja template is located.

  • template_file (str) – The name of the Jinja template file.

  • render_vars (dict) – Dictionary containing the definitions of all the variables present in the Jinja template.

Returns:

output_text – Content of the rendered file.

Return type:

str

This function receives three key arguments describing where to find the template and what is the corresponding content. It then returns the content of the rendered file in a variable (output_text), that will be printed in a file by the main script control_launcher.py.

5.4.4. Simple function model

In essence, the rendering functions have a pretty simple structure: their task is to define the name of each Jinja template and define the needed content for that template (stored in <file>_render_vars). Here is a basic rendering function model:

def my_rendering_function(clusters_cfg:dict, config:dict, system:dict, data:dict, job_specs:dict, misc:dict):

   # Define the names of the templates

   template_param = "name_of_parameters_template"
   template_script = "name_of_job_script_template"

   # Define the names of the rendered files

   rendered_param = "name_of_created_parameters_file"
   rendered_script = "name_of_created_job_script_file"

   # Initialize the dictionary that will be returned by the function

   rendered_content = {}

   # Render the template for the parameters file

   param_render_vars = {
      "jinja_variable1" : value,
      "jinja_variable2" : value,
      "jinja_variable3" : value,
      ...
   }

   rendered_content[rendered_param] = jinja_render(misc['templates_dir'], template_param, param_render_vars)

   # Render the template for the job script

   script_render_vars = {
      "jinja_variable1" : value,
      "jinja_variable2" : value,
      "jinja_variable3" : value,
      ...
   }

   rendered_content[rendered_script] = jinja_render(misc['templates_dir'], template_script, script_render_vars)

   # Return the content of the rendered files and the name of the rendered job script

   return rendered_content, rendered_script

A basic example of a rendering function is presented at the end of this section. Note however that some additional steps might be required depending on each specific case.

5.4.5. Calling your rendering function

The rendering function that will be called by CONTROL LAUNCHER is the one associated with the rendering_function YAML key defined in the clusters configuration file:

mycluster:
   profiles:
      myprofile1:
         rendering_function: name-of-rendering-function
      myprofile2:
         rendering_function: name-of-rendering-function

where mycluster corresponds to the name of your cluster (given as a command line argument) while myprofile1 and myprofile2 are the names of the profiles you want to use. This way, a different rendering function can be assigned to each profile.

5.4.6. Example of a rendering function

Todo

COMING SOON

5.4.6.1. Review of the template and configuration files

Before defining the function, we need to review our different files:

First, we have the parameters file template:

sample_param.nml.jinja
&GENERAL
  systeme = "{{ source_name }}",
  processus = "{{ processus }}"                ! Nature du processus : [OPC] Contrôle optimal, [OPM] Contrôle a contraintes multiples ; [PCP] Post-contrôle avec pulse [PCL] Post-contrôle libre
/

&OPTIONS                         ! [1] oui [0] non
  op_ch = 2,                     ! Activer l'écriture des champs ([2] à chaque itération)
  op_ver = 0,                    ! Activer les fichiers de vérification
  op_mat = 0,                    ! Activer l'impression des matrices (routine sqrtm) ([2] désactiver le calcul de la fidélité de Uhlmann) 
/

&DATA_FILES
  cheminE = "{{ energies_file_path }}",                 ! Hartree
  cheminMD = "{{ momdip_e_path }}",                     ! Unités atomiques
  eti = "{{ init_file_path }}",
  etf = "{{ final_file_path }}",
  projector = "{{ proj_file_path }}",                  ! A la place d'un état final spécifique
/

&CONTROL
  niter = {{ niter }},                          ! Nombre d'itérations
  seuil = {{ threshold }},                      ! Seuil de recouvrement pour le contrôle
  dt = {{ dt }},                                ! Pas de temps en unités atomiques
  source = "{{ source }}"                       ! Reprise d'un champ (OPC, PCP) ou lecture d'états init (PCL). Res/$système/$operation/...  (PCP: $nomcalcul/Pulse/$nomfichierpulse -- PCL: [H] Psii_RI$n , [L] chif$n !ne pas mettre le $n)
/

&OPC
  nstep = {{ nstep }},                          ! Nombre de pas de temps
  fnelle = "NF",                                  ! Fonctionelle (AF ou NF) pour les cas multicible
  alpha0 = {{ alpha0 }},                        ! Contrainte sur l'amplitude du champ (alpha0, nulle si =1)
  ndump = {{ ndump }}                           ! Nombre d'iterations avant ecriture du champ 
/

&OPM_PULSE
  numberofpixels = {{ numberofpixels }},
  inputenergy = {{ inputenergy }},                     ! in microjoule per cm2
  widthhalfmax = {{ widthhalfmax }},                   ! in spectrum, in cm-1
  omegazero = {{ omegazero }}                          ! in spectrum, in cm-1
/

Then, the job script template:

sample_qoctra_job.sh.jinja
#!/bin/bash

#SBATCH --output=slurm_output.log
#SBATCH --job-name={{ source_name }}_{{ transition }}_{{ config_name }}
#SBATCH --mail-user={{ user_email }}
#SBATCH --mail-type={{ mail_type }}
#SBATCH --time={{ job_walltime }}
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu={{ job_memory }}
{% if partition != None %}
#SBATCH --partition={{ partition }}
{% endif %}

QOCT_RA_DIR="${CECIHOME}/QOCT-GRAD"

echo -e ">>> Start of QOCT-GRAD compilation"

{{ set_env }}
gfortran -lopenblas -O3 -ffast-math -funroll-loops -fwhole-program -flto -fexternal-blas -fdefault-integer-8 -m64 $QOCT_RA_DIR/Sub/Fortran/mymod.f $QOCT_RA_DIR/Controle.f90 -o Controle.out
rm mymod.mod  

echo -e ">>> End of QOCT-GRAD compilation"

echo -e "\n================= QOCT-GRAD execution begins now =================="

./Controle.out {{ rendered_param }} $QOCT_RA_DIR

echo -e "\n================= QOCT-GRAD execution ends now =================="

Third, the configuration file:

config.yml
user_email: your@email.com
mail_type: FAIL
control:
  dt: 7.d0                                      # Duration of a time step (a.u. of time)
  niter: 3000                                   # Number of iterations
  threshold: 0.9999d0                           # Convergence threshold for the fidelity
  alpha0: 250.0d0                               # Constraint on the field amplitude
  ndump: 30                                     # Number of iterations between each writing of the field
  numberofpixels: 128                           # Number of pixels (pixel width = 2 / total time)

Finally, let’s consider that we have the following clusters configuration file:

clusters.yml
lemaitre3:
  submit_command: sbatch
  profiles:
    qchem_gt_opm:
      parsing_function: qchem_tddft
      transition_function: proj_ground_to_triplet
      rendering_function: sample_qoctra_render
      set_env: module load OpenBLAS/0.3.7-GCC-8.3.0                  
      job_scales:
      - 
        label: small
        scale_limit: 20
        time: 1-00:00:00
        memory: 2000 # in MB
      - 
        label: medium
        scale_limit: 50
        time: 2-00:00:00
        memory: 2500 # in MB
      - 
        label: big
        scale_limit: 100
        time: 5-00:00:00
        memory: 3000 # in MB

where lemaitre3 is the name of our cluster.

5.4.6.2. Function definition

Todo

COMING SOON