Condor SOAR Notes

• Condor's SOAR Stat Submit Site
• Condor's SOAR Submit Site
• SOAR Manual
• DAGMan Manual Pages
• How To Use SOAR Drop Boxes
• How To Manage Stat's SOAR Server
• New Generic & qtlhot Notes

How to Use SOAR Drop Boxes

• Drop box tasks are initiated every half-hour.

  Initial Setup of User Project Area

• Each user has a run area, typically at /scratch/users/login/rundata, with login replaced by the user's login. This area is set up by the SOAR administrator. All folders from the login name down are owned by the user, and should be made readable by the statsoar group and/or the world. Note that only users with valid logins to simon.stat.wisc.edu will be able to see these folders.
• A project requires two folders. The project folder (replaced with the name of the project, e.g. qtlnet) has the data to be used for the project, while the project_objs folder has any code objects that are needed. For a predefined project such as qtlnet, this may be empty. However, the user can override default code objects by placing ones of the same name here.

  % mkdir /scratch/users/login/rundata/project
  % mkdir /scratch/users/login/rundata/project_objs
  

  Submitting Runs

  • To submit a run via the "drop box" technology, you need to create a folder under the project folder. It is best to do this in your usual work area and then copy to the /scratch/users/login/rundata/project folder once it is ready to go. Why?
    1. You maintain a clean copy in a place you can back up.
    2. You don't mistakenly submit a partial run by crossing a half-hour mark while creating your materials.
  • Create a folder within the project called dataset_XXX (always keep dataset_ as front part of folder name; use XXX to give a name that you will remember).
  • The dataset_XXX folder should contain:
    1. Common dataset objects to be used across related runs, in this case something like cross.RData.
    2. Separate folders for each run (such as run1, run2.
  • The run folder contains files for a specific run, typically a flat file with options and possibly other dataset objects.
  • Example for qtlnet in /scratch/users/yandell/rundata/qtlnet/:
    1. Dataset folder dataset_test contains cross.RData, an R data object with an R/qtl cross object containing 13 phenotypes.
    2. Run folder dataset_test/qtlnet1 contains the file params.txt.
    3. The first line of params.txt contains the line nruns = 100 ;, which is parsed by a PERL script in the SOAR job for qtlnet to find out how many MCMC runs to perform in Phase 4. You can change the number of runs from 100, but keep the exact format.
    4. Other lines should contain valid R code designed to assign values to qtlnet arguments. The pheno.col can be entered as numbers or character strings; addcov should be a list.
  • Space/Time Tradeoffs: Short runs of 1 minute are possible, but there is some overhead. Best to aim for 1 hour runs for economies of submit node access.

  Monitoring Runs

• Once a job has run and failed or worked, changing it will not have it run again. It is remembered..... There are several ways to rerun a test:
  • Simply rename the folder to one it has not seen yet.
  • Create a qtlnet_objs/objconfig file and place the test in the whitelist section.
  • Do a --kind=reset and rerun all.
• Drop box in qtlnet
• Error detection/run monitoring: Check http://soar.stat.wisc.edu under the run (qtlnet) and find the latest number. Clicking on the Profile will show a histogram of active or completed tasks for this job; Report shows a text file summary that is not too helpful; Results eventually has the completed results. The number has the running files and can be checked periodically. In it you will find:
  • movieruns.dag.dagman.out: running log of all jobs.
  • movieruns.dag.dagman.rescue001: info on why you did not get through the run.
  • dataset_XXXX-jobn/: folder with more details of subjobs. See the *dagman.out file for complete log. Check individual .log, .err and .out files.

How to Manage Stat's SOAR Server

• Initial setup of user areas (can set up folder as statsoar, but need SU to chown)

  % mkdir /scratch/yandell/rundata
  % chown yandell /scratch/yandell/rundata
  

• StatSoar File Organization
  • There are several locations that are of particular interest, primarily for the "statsoar" user. Bill Taylor has set up aliases to get to these. Below they are surrounded by **'s. For instance *sources*/qtlnet/v3 translates as

    % sources
    % cd qtlnet/v3
    

  • SOAR is set to allow things to be splattered across different file systems (e.g. for sources, condorruns, and results), with the interface home at one location.

    *run*     /scratch/htdocs
    *control* /p/stat/soar/control
    *sources* /p/stat/soar/sources
    

    For soar.stat, all are on soar.stat.wisc.edu for now. Some more detail on these can be found at the end of this document.
  • The bits and pieces are mostly in the SOAR Manual.
  • See *run*/.. for files used in automatic setup: continuous.cron (per_runs), checkprogress.cron (per_plotsandreports).
  • SOAR checks in user drop box first, but then use *sources*/qtlnet/v3/code for other files.
• There are several ways to reset jobs (see drop boxes above). In addition, in /scratch/htdocs/tracking/qtlnet/v2/DONE you could remove the test. There is another way to create a file with a list of tests(min 1), but I do not understand that yet.
• Prepare R Libraries for Submit: If only one library, then place in project_objs folder in the sources folder. If more than one, the use the Prepare R Library PERL script (in prep from BT) to catenate in desired order.
  • The *sources*/qtlnet/v3 has three directories
    • code: code supplied by user for project
    • jobs: job code generated by SOAR to run project
    • rlibs: R libraries from user (name_version.tar.gz) and packaged RLIBS.tar.gz
  • To create the libaries, use the script to pass a comma separated list of libraries in the order of their dependencies.

    % sources
    % cd qtlnet/v3/rlibs
    % ./PrepareRLibs.pl qtl_1.16-6.tar.gz,qtlnet_0.9.4.tar.gz
    

    Then copy new RLIBS.tar.gz to code directory
• Web pages: Can set .htaccess in *source*/project/jobs. In /scratch/htdocs is the file projects.webinput. Add any projects you want in a similar manner here and then type the following "control" command from the control location as the statsoar user:

  % control
  % ./control --project=any --version=any --kind=newproject
  

  As long as it sees any project version pair(real or not) this will regenerate the index file and quit.

• Control Scripts (see Control Scripts examples in SOAR Manual)
  • Start a project that is read to run: The simplest forced run is

    % ./control --project=qtlhot --version=v1 --kind=new
    

    This type of execution is done every half hour, but statsoar can start at will with this command.
  • Duplicate a project

    % control
    % ./control --preproject=natehighres --preversion=v1 --project=DanLewisArabidopsis --version=v1 --kind=install
    

  • See --kind=new entries in per_runs for continuous.cron
  • Consider --limit=2 for tests to restrict number of jobs
• When you want to remove a run (say 29) so that it can be rerun use

  % control
  % ./soar_rm --project=xxxxxxxxx --env=29
  

  This adjusts the RUNNING to not list this one so it is now neither DONE nor RUNNING.

  cp -r /scratch/htdocs/imageruns/qtlnet /scratch/users/yandell/rundata
  

• % control ## master scripts (only operator should modify)
  • % xemacs fsconfig ## add lines for specific projects after IMAGERUNS
  • % ./control ... ## see Control Examples
• % run; cd .. ## run directory
  • % ls imageruns ## default dataset area
  • % ls condorruns ## place where condor run results are put
  • continuous.cron uses per_runs (put ./condor projects here with kind=new to have continuous checking)
  • checkprogress.cron uses per_plotsandreports
• % sources ## source code for projects and versions
  • % ls qtlnet/code ## code from user for project
  • % ls qtlnet/jobs ## SOAR PERL scripts to process code
    • phaseN.template ## construct actual submit for job
    • phaseN.pl ## remote job call using PERL

New Generic & qtlhot Notes

rundata: Look at stage_qtlhot, qtlhotGENERIC and qtlhot. The stage_qtlhot directory has stage.pl to run the spawned qtlhot children. The qtlhotGENERIC/NEWDATA says to look in /home2/bt/rundata for project qtlhot. In qtlhot, find dataset_blah.

SOAR/sources: Folder qtlhot/v1 has code and jobs directories. The perl scripts in jobs were curated by bt to set up phase1,phase2,phase3. The postjobphase1.pl spawns nruns (read from params.txt file) of dataset_blah-n, n=1:nruns, one for each phase2 run. The phase2 runs each spawn n.split (read from groups.txt file created in phase1, initially read from params.txt in phase1) sub-jobs. Also look at make_job_{dag_text,submits}.pl for the flow of the template.

In order to set up a generic 3-phase run for Aimee, I think I can copy qtlhot and make a few simple mods. If nruns is set to 1, then we can use most of the code. Then different R libraries and code could be dropped in dataset_job files in the qtlruns project. An idea.