Sun NG1 - Revision history

Tdhaene: /* Using it */

2008-06-13T09:59:31Z

Using it

← Older revision		Revision as of 09:59, 13 June 2008
Line 149:		Line 149:
	== Using it ==		== Using it ==

	Put all the above files in one folder, e.g. ~/SUMO_config. Construct your config file with different runs in the same folder. Run		Put all the above files in one folder, e.g. <code>~/SUMO_config</code>. Construct your config file with different runs in the same folder. Run

	<code>		<code>

Tdhaene: /* The Full config file */

2008-06-13T09:58:23Z

The Full config file

← Older revision		Revision as of 09:58, 13 June 2008
Line 120:		Line 120:
	</pre></code>		</pre></code>

	== The ~~Full~~ config file ==		== The full config file ==

	Just make a complete complex file with more than one run. If you want separate log files, replace the logging line by		Just make a complete complex file with more than one run. If you want separate log files, replace the logging line by

Tdhaene at 09:56, 13 June 2008

2008-06-13T09:56:59Z

← Older revision		Revision as of 09:56, 13 June 2008
Line 1:		Line 1:
	The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the [[Config:SampleEvaluator#~~begrid~~\|distributed sample evaluators]]!		The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the [[Config:SampleEvaluator#calcua\|distributed sample evaluators]]!

	== The skeleton job script ==		== The skeleton job script ==

Tdhaene at 09:56, 13 June 2008

2008-06-13T09:56:36Z

← Older revision		Revision as of 09:56, 13 June 2008
Line 1:		Line 1:
	The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the [[~~http://sumowiki.intec.ugent.be/index.php/~~Config:SampleEvaluator#begrid\|distributed sample evaluators]]!		The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the [[Config:SampleEvaluator#begrid\|distributed sample evaluators]]!

	== The skeleton job script ==		== The skeleton job script ==

Tdhaene at 09:56, 13 June 2008

2008-06-13T09:56:24Z

← Older revision		Revision as of 09:56, 13 June 2008
Line 1:		Line 1:
	The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the distributed sample evaluators!		The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the [[http://sumowiki.intec.ugent.be/index.php/Config:SampleEvaluator#begrid\|distributed sample evaluators]]!

	== The skeleton job script ==		== The skeleton job script ==

Dgorissen at 13:37, 6 February 2008

2008-02-06T13:37:44Z

New page

The scripts on this page help you to run multiple toolbox instances in parallel on a SGE cluster. Note this is not the same as using the distributed sample evaluators!

== The skeleton job script ==

This script will not run! However, it is used by the split script. Adjust the MATLABDIR and SUMO variables to point to the toolbox installation and the matlab installation. Replace the -USER- tag by your username.

<code><pre>
FILE: SUMOJob.sh

#!/bin/csh
# Start the job in the current working directory
#$ -cwd

#$ -S /bin/bash

# Specify files for the standard output and error

MATLABDIR=/storeA/soft/matlab-7.3
sumo=/storeA/users/wohe/trunk

#$ -N SUMOToolbox
#$ -o -NAME-.$JOB_ID.out
#$ -e -NAME-.$JOB_ID.err
#$ -q normalprio

# Let's run matlab now
echo `uname -a`
ssh -L 27000:143.129.75.6:27000 -L 28000:143.129.75.6:28000 -N babbage &

${MATLABDIR}/bin/matlab -nodisplay -r "cd ${sumo};startup;go('-CONFIG-');exit;";

pkill -u -USER-
</pre></code>

== The split script ==

This is the main device for splitting a configuration file into different config files for each run.

<code><pre>
FILE: split.py

#!/usr/bin/python
import string
import os
import sys

fname = sys.argv[1]
print 'File used : ', fname

if len(sys.argv) > 2:
replicates = int( sys.argv[2] )
else:
replicates = 4

f = open( fname, "r")
lines = f.readlines()
f.close()

f = open( 'SUMOJob.sh', 'r' )
joblines = f.readlines()
f.close()

start = filter( lambda x: lines[x].find( '<Run' ) >= 0, range(len(lines)) );
stop = filter( lambda x: lines[x].find( '</Run' ) >= 0, range(len(lines)) );

plan1 = start[0]-1
plan2 = filter( lambda x: lines[x].find( '</Plan' ) >= 0, range(len(lines)) );

if len(plan2) != 1 or len(start) != len(stop):
print 'error'
sys.exit(-1)

plan2 = plan2[0]

def putLines( f, l ):
for line in l:
f.write( line )

def cut( delim, f, l ):
return string.split( l, delim )[f-1]

def escape( str ):
return str.replace( '/', '\\/' ).replace( '-', '\\-' )

folder = cut( ".", 1, fname )
os.system( 'rm -Rf %s ; mkdir %s' % (folder,folder) )

currmap = os.getcwd()
counter = 1

os.system( 'if [[ ! -e output ]]; then mkdir output; fi' )

for i in range(len(start)):
# Build job shell files
for j in range(1,replicates+1):

# Build xml config file
runname = cut( "\"", 2, lines[start[i]] )

fname = "%s/Config_%04d.xml" % (folder,counter)
print 'Generating', fname
f = open( fname, "w" )
putLines( f, lines[:plan1+1] );
putLines( f, lines[start[i]:stop[i]+1] )
putLines( f, lines[plan2:] );
f.close()

# Patch output log file
os.system( 'cat %s | sed \'s/\\-LOGFILE\\-/%s\\/output\\/%s_rep%d_LOG.txt/\' > tmp | mv tmp %s' % (escape(fname),escape(currmap),escape(runname),j,escape(fname)) )

jobfile = "%s/Job_%04d.sh" % (folder,counter)
print ' - Job file:', jobfile
f = open( jobfile, 'w' )
for line in joblines:
f.write( line.replace( '-NAME-', '%s/output/%s_rep%d' % (currmap,runname,j) ).replace( '-CONFIG-', '%s/%s' % (currmap,fname) ) )
f.close()

counter = counter + 1
</pre></code>

== The Full config file ==

Just make a complete complex file with more than one run. If you want separate log files, replace the logging line by

<code><pre>
<Option key="java.util.logging.FileHandler.pattern" value="-LOGFILE-"/>
</pre></code>

Make sure no runs are commented out, and make sure no opening <run> tags are on the same line as closing </run> tags.

== The batch submit script ==

Use this script to submit the splitted job files.

<code><pre>
FILE: submit.sh

#!/bin/bash

path=`pwd`/$1
ls $path/*.sh | while read f
do
echo "Submitting $f"
qsub $f;
done
</pre></code>

== Using it ==

Put all the above files in one folder, e.g. ~/SUMO_config. Construct your config file with different runs in the same folder. Run

<code>
./split.py <configfile> <replicates>
</code>

A folder called after the configfile is generated, containing .sh scripts and .xml config files for each execution.

Now run <code>./submit <foldername></code> to submit all jobs in that folder at once.

Output will appear in
* The output directory of your toolbox installation
* The output directory under the directory where the scripts are installed. Please note that the split.py script CLEARS the output directory when run.