Here are the updated instruction of generating Monte Carlo for SpinQuest simulations and analysis

• Login to Rivanna (https://ood.hpc.virginia.edu/pun/sys/dashboard)
• Create a folder to submit your jobs from on /project/ptgroup/ or /project/UVA-Spin
• The scripts to generate Monte Carlo (MC) events on Rivanna are in: 

  /project/ptgroup/work/MC_Generation

  You can make more and store the generating scripts here but label them in a way that is easy to understand.  The present naming convention is <Exp>_<channel>_<vertex origin>
  • Option for <Exp> are 906 or 1039.  The Geant4 geometry is different for the setup of E906 and E1039, so for SpinQuest make sure you are using the E1039 geometry.
  • Option for <channel> are : Drell-Yan (DY), Jpsi (JPsi), Psi' (Psip) Pion background (Pion), Kaon background (Kaon), Random-multi muons (MultiMuon), Single Muons + or- (SingMup/m)
  • Option for <Vertex origin> : target (Target), Beam Dump (Dump), Everything that seen by the beam (All), The gap between the target and dump (TargetDumpGap), and arbitrary vertex origin (Manual, should specify numbers such as x,y,z→25,25,100 would be a target size with diameter 25cm and 100cm long.
• You can copy any MC scripts to your directory to submit.  The scripts to generate Monte Carlo (MC) events on Rivanna are in: 

  /project/ptgroup/work/MC_Generation

  Example of copying: cd /project/ptgroup/YourFolder

  rsync -av --exclude scratch /project/ptgroup/MC_Generation/DY_Target_script .

  
  

  We will keep track of all MC generation scripts so we can all see how a particular MC batch was produced.  Details of each one should be kept in ReadMe in that directory.
• Other MC configurations you make that are not included in this directory you should copy over and make a note about so other people can use it as well.
• To summit you jobs to produce the MC navigate to the directory where you want to run the script.
• Setup the environment using: source /project/ptgroup/spinquest/this-e1039.sh  command
  

• Submit your job using the following command.
  $./jobscript.sh<Outpu{t}_{f}olde{r}_{n}ame>\; <\mathrm{Number\; of}{}_{j}obs>\; <\mathrm{Number\; of\; Events\; per\; Job}>\; For\; example:$./jobscript.sh DY_Target_1M  100  10000       (Here we need to generate ~1M events. The accepted events depend on the channel). Strongly recommended not to exceed 10K events per job.

            The jobscript.sh should look like this:

#!/bin/bash

dir_macros=$(dirname $(readlink -f $BASH_SOURCE))
jobname=$1
njobs=$2
nevents=$3
echo "njobs=$njobs"
echo "nevents=$nevents"
work=/scratch/$USER/MC/$jobname
mkdir -p $work
chmod -R 01755 $work
cd $dir_macros

for (( id=1; id<=$njobs; id++ ))
do 
  mkdir -p $work/$id/
  chmod -R 01755 $work/$id
  cd $work/$id/
  cp $dir_macros/*.C .
  cp $dir_macros/*.cfg .
  cp $dir_macros/*.txt .
  cp $dir_macros/*.slurm .
  sed -i "s/1234/$nevents/" Fun4Sim.C
  echo "submitting job number = $id"
  sbatch grid.slurm

   Which will submit the job according to the grid.slurm script:

#!/bin/sh
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=6:00:00
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --partition=standard
#SBATCH -A spinquest
export DISPLAY=:0.0
root -l Fun4Sim.C
root -l filter.C

• This is the slurm script for Rivanna (https://www.rc.virginia.edu/userinfo/hpc/slurm/)
• sbatch -A gives the Rivanna allocation.  Both spinquest and spinquest_standard should be available.
• You can use "squeue -u user", to check the status of your jobs (or use the "Active Jobs" tab on your UVA OpenOnDemand web page: https://ood.hpc.virginia.edu/pun/sys/dashboard/activejobs?jobcluster=all&jobfilter=user).
• Navigate to "/project/ptgroup/script"  and then implement the following command (this script by-default will skip bad-files/corrupted-files/etc.)
  $ ./merge_mc_prod.sh /scratch/<your_MC_output_file_location>
  
  This will create a folder inside /project/ptgrpup/spinquest/MC_merge_files" with the same name as your MC_output_folder
• Copy the "merged_trackQA_v2.root" file from the folder that you've newly generated by ./merge_mc_prod.sh and copy it to "/project/ptgroup/spinquest/MC_storage" location. Rename the file into <Channel>_<Vertex Origin>_<Number of accepted event>.root. for example: JPsi_Dump_300K.root