virgo cluster

The batch farm runs on the virgo.hpc.gsi.de cluster. The batch jobs are managed by the SLURM (SL) scheduler. The documentation on the cluster/SLURM can be found here.
New Hades users have to be added to the users of account hades to be able to run jobs on the farm. The list of hades users is maintained by J.Markert@gsi.de

Some rules to work with virgo.hpc.gsi.de:

• Files are written to /lustre filesystem. Each Hades user owns a directory /lustre/hades/user/${USER} to  work on the farm (NO BACKUP!)
• SLURM does not support other files systems than /lustre. batch
  scripts can not use user's home dir nor any other filesystem
• The hadessoftware is distributed to the batchfarm via
  /cvmfs/hades.gsi.de/ (debian8) or /cvmfs/hadessoft.gsi.de (debian10)
  The /cvmfs/hades.gsi.de or /cvmfs/hadessoft.gsi.de are available on desktop machines dependening on the OS version
• The batch jobs can be submited to the farm from the virgo.hpc.gsi.de
  cluster. This machines provides our software, the user ${HOME} and
  a file sytem mount to /lustre. You can compile and test (run) your
  programs here.
• A set of example batch scripts for Pluto, UrQmd, HGeant, DSTs and user
  analysis you can retrieve from
  svn checkout https://subversion.gsi.de/hades/hydra2/trunk/scripts/batch/GE
  The folders contain sendScript_SL.sh+jobScript_SL.sh (SL).
  The general concept is to work with file lists as input to the sendScript, which
  takes care about the sync from user homedir to the submission dir on
  /lustre. The files in the list are splitted automatically to job arrays to
  minimize the load on the scheduler. The sendScript finally calls the sbatch
  command of SLURM to submit the job. The jobScript is the part which
  runs on the batch nodes.

Working on the batch farm of GSI:

virgo cluster documentation:

https://hpc.gsi.de/virgo/user-guide/storage.html

ssh key autentication

https://hpc.gsi.de/virgo/user-guide/access/key-authentication.html

Since 1st of September 2021 we have two software environments
available:

debian10: neweset software releases compiled with root 6.24.02 and gcc 8.3
         use this for current beam mar19 analysis of dst files. The login to
         the farm will use vae23.hpc.gsi.de (debian10,ROOT6) and no special action
         for handling the container environment with singularity is needed.
         It works as the not anymore vailable virgo-debian8.hpc.gsi.de
         
debian8: old software releases compiled with root 5.34.34 and gcc 4.9.2
        This covers all versions up to hydra2-5.6a used for dst productions
        of apr12,aug14,jul14, mar19 (gen5) pre 1st of September 2021
        use this to get exact same behaviour as before the switch and do
        not want to change any software. To make the environment work
        some handling of the singularity options are needed to start and
        submit jobs using this container environment. It will be described
        below.

#############################################################

virgo cluster usage information:
           virgo.hpc.gsi.de(bare bone to start your container), 
           vae23.gsi.de (debian10 container started at login)

Problems and tips:

•  if your login to virgo.hpc.gsi.de
   does not work although you have/created ssh keys
   (https://hpc.gsi.de/virgo/access/key_authentication.html),
   try to cleanup .ssh/authorized_keys from other
   keys which need other credentials and might cause
   problems.
•  ksh does not work with the login, 
   use bash (accounts-service@gsi.de).
   ksh is not installed at virgo and will
   lead to permission denied statements without further
   explanation. accounts-service@gsi.de will
   be responsable to change your login shell.
   It will take a while to see the changes, syncs
   are performed once per day at 22:00.

•  X does not work at virgo at the moment :
   a. use lx-pool.gsi.de (or any desktop machine) to look at /lustre output
      use sshfs to mount /lustre to any linux machine which
      has no mount of /lustre already. 
      To mount /lustre us

  sshfs user@virgo.hpc.gsi.de:/lustre mymountpoint
  sshfs user@lustre.hpc.gsi.de:/lustre mymountpoint

         vae23.hpc.gsi.de will not work with sshfs
         since the machine will be closed after the sshfs command has been returned.

•  our sendScript_SL.sh for batch submission before using singularity containers
   needs a small modification for virgo:

  # from inside virgo container
      command="--array=1-${stop} ${resources} -D ${submissiondir}  --      output=${pathoutputlog}/slurm-%A_%a.out -- ${jobscript} ${submissiondir}/${jobarrayFile} ${pathoutputlog} ${arrayoffset}"
     for virgo there is an additional " -- " between slurm options and 
      the user scripts+parameters of the script which should be started
      at the farm. This is needed since slurm seperates from the container
      which is started. The container version is choosen from the submit host
      automatically.
  #############################################################

debian10 container login:

ssh username@vae23.hpc.gsi.de            (virgo)

This command will start a container based instance
of gsi debian10.

debian8 container login:

working environment debian8:
1. login   ssh username@virgo.hpc.gsi.de.   
2. start debian8 container:   
 

 . start_debian8.sh

start_debian8.sh:
//-------------------------------
export SSHD_CONTAINER_DEFAULT="/cvmfs/vae.gsi.de/debian8/containers/user_container-production.sif"
export SSHD_CONTAINER_OPTIONS="--bind /etc/slurm,/var/run/munge,/var/spool/slurm,/var/lib/sss/pipes/nss,/cvmfs/vae.gsi.de,/cvmfs/hadessoft.gsi.de/install/debian8/install:/cvmfs/hades.gsi.de/install,/cvmfs/hadessoft.gsi.de/param:/cvmfs/hades.gsi.de/param,/cvmfs/hadessoft.gsi.de/install/debian8/oracle:/cvmfs/it.gsi.de/oracle"
shell=$(getent passwd $USER | cut -d : -f 7)
STARTUP_COMMAND=$(cat << EOF
   srun() { srun-nowrap --singularity-no-bind-defaults "\$@"; }
   sbatch() { sbatch-nowrap --singularity-no-bind-defaults "\$@"; }
   export -f srun sbatch
   $shell -l
EOF
)
export SINGULARITYENV_PS1="\u@\h:\w > "
export SINGULARITYENV_SLURM_SINGULARITY_CONTAINER=$SSHD_CONTAINER_DEFAULT
test -f /etc/motd && cat /etc/motd
echo Container launched: $(realpath $SSHD_CONTAINER_DEFAULT)
exec singularity exec $SSHD_CONTAINER_OPTIONS $SSHD_CONTAINER_DEFAULT $shell -c "$STARTUP_COMMAND"
//-------------------------------
 

This environment will allow to compile and use code. SL_mon.pl
and access the SLURM commands will not work on debian8.
virgo.hpc.gsi.de + vae23.hpc.gsi.de allow to use SL_mon.pl+SLURM

3. our sendScript_SL.sh for batch submission works
  for the vae23.hpc.gsi.de login. NO ADDITIONAL wrap.sh NEEDED
  (see below)!
------------------------------------------------------------

submission of batchjobs for debian8 virgo3:

1. login to virgo.hpc.gsi.de
2. modifiy your sendScript_SL.sh to use wrap.sh
  to start the debian8 container on the farm.

wrap.sh:
//-------------------------------
#!/bin/bash
jobscript=$1
jobarrayFile=$2
pathoutputlog=$3
arrayoffset=$4
singularity exec \
-B /cvmfs/hadessoft.gsi.de/install/debian8/install:/cvmfs/hades.gsi.de/install \
-B /cvmfs/hadessoft.gsi.de/param:/cvmfs/hades.gsi.de/param \
-B /cvmfs/hadessoft.gsi.de/install/debian8/oracle:/cvmfs/it.gsi.de/oracle \
-B /lustre \
/cvmfs/vae.gsi.de/debian8/containers/user_container-production.sif  ${jobscript} ${jobarrayFile} ${pathoutputlog} ${arrayoffset}
//-------------------------------

in sendScrip_SL.sh in the lower part where the sbatch command is build:
    
    #virgo bare bone submit using wrap.sh 
    wrap=./wrap.sh
    command="--array=1-${stop} ${resources} -D ${submissiondir}  --output=${pathoutputlog}/slurm-%A_%a.out -- ${wrap} ${jobscript} ${submissiondir}/${jobarrayFile} ${pathoutputlog} ${arrayoffset}"
 

SLURM tips:

The most relevant commands to work with SL:

sbatch   : sbatch  submits  a  batch script to SLURM.
squeue   : used to view job and job step information for jobs managed by SLURM.
scancel  : used to signal or cancel jobs, job arrays or job steps.
sinfo    : used to view partition and node information for a system running SLURM.
sreport  : used to generate reports of job usage and cluster utilization for
		   SLURM jobs saved to the SLURM Database.
scontrol : used  to  view  or modify Slurm configuration including: job, 
           job step, node, partition,reservation, and overall system configuration.
 
Examples:
squeue  -u  user             : show all jobs of user
squeue -t  R                 : show jobs in a certain state (PENDING (PD),
                               RUNNING (R), SUSPENDED (S),COMPLETING (CG),
                               COMPLETED (CD), CONFIGURING (CF),
                               CANCELLED (CA),FAILED (F), TIMEOUT (TO),
                               PREEMPTED (PR), BOOT_FAIL (BF) ,
                               NODE_FAIL (NF) and SPECIAL_EXIT (SE))
scancel  -u user             : cancel all jobs of user user
scancel jobid                : cancel job with jobid
scancel -t PD -u <username>  : cancel all pending jobs of a user
scontrol show job -d <jobid> : show detailed info about a job
scontrol hold <jobid>        : suspend a job
scontrol resume <jobid>      : resume a suspended job

The Hades Subversion (svn , http://subversion.tigris.org/) repositories are located at the GSI web-server. Some information about subversion at GSI can be found at http://wiki.gsi.de/Linux/SubVersion The access authentification uses the ORACLE data base of GSI. Read permission is granted anonymously. For commiting code you have to have an account at http://www-oracle.gsi.de/ . This is the same account as used for the documentation of working time or the radiation savety. You do not need to have GSI linux or windows account to get an user account. The user name has to added to the subversion access management. Mail your user name and which directries you want to work with to j.markert@gsi.de.

Currently available repositories:

• https://subversion.gsi.de/hades/hydra : All history up to last stable version v8_21 (read permission only)
• https://subversion.gsi.de/hades/hgeant : All history up to last stable version v8_21 (read permission only)
• https://subversion.gsi.de/hades/hydraTrans : Transition repository to restructure all libraries and prepare for next runs from 2010 on (closed since 22.06.2011, only read permission)
• https://subversion.gsi.de/hades/hydra2 : Final repository after transition period
• https://subversion.gsi.de/hades/hgeant2 : Final repository after transition period
• https://subversion.gsi.de/hades/pluto : Final (restricted access)
• https://subversion.gsi.de/hades/fwdetsn : transition repoitory for Forward Detector (FWDET) development
• https://subversion.gsi.de/hades/publications : repository for publications (restricted access)
  (Table of content on the web)

Web-front ends of the available repositories:

The GSI IT provides redmine , a web-frontend for the subversion repositories, running on the apache webserver. This web-frontend replaces our old CVS frontend. 

hades svn redmine

Basic use of Subversion:

For the documentaion of subversion see http://svnbook.red-bean.com/

 ---------------------------------------------------------------------------------
# checkout a repository

# get full repository from trunk (main branch) into a folder hydraTrans
svn co https://subversion.gsi.de/hades/hydraTrans/trunk hydraTrans

# get a directory from the repository trunk (main branch) into a folder hydraTrans
svn co https://subversion.gsi.de/hades/hydraTrans/trunk/mdc hydraTrans/mdc
---------------------------------------------------------------------------------
# view all commands
svn help

# view help on commands
svn help status
---------------------------------------------------------------------------------
# most usefull commands
# to work on the local working copy

[file] means filename is optional. In this case the commands
apply to all files in the current directory

svn stat    [file]                                        // show local changes (stat=status)
    stat -u [file]                                        // show local changes and changes on the server (-u == in update mode)
svn diff [file]                                           // show modification of a file against a revision
svn add file                                              // schedule a new file for adding. Needs commit afterwards to send it to the repository
svn update [file]                                         // update file to newest revision
svn commit -m "your comment" [file]                       // send file [or all modified files] to repository (requires access permissions). takes the user name from checkout log
svn --username yourname commit -m "your comment" [file]   // send file [or all modified files] as agiven svn user to repository (requires access permissions).
                                                          // helpful to commit from a checkout dir of another user



tkdiff file                                               // show graphical diff of file to svn base revision
tkdiff file -r head                                       // show graphical diff of file to newest svn head revision