Contents
Skip to end of metadata
Go to start of metadata
Table of Contents

Overview

This section covers how to use OASIS CVMFS system to run a real application like Octave or MatLab statistical package. For this example, we'll use Octave although Matlab can be substituted without any changes in the code. This example will go through using Matlab to generate a random matrix, doing some simple matrix operations and then calculate the eigenvectors of the matrix.

Setup

First, let's create a work directory. As always, you can use tutorial instead.

$ mkdir -p tutorial-octave/log
$ cd tutorial-octave

Testing Octave

Since Octave is installed into CVMFS, it's not available in the normal system paths. We'll need to set up those paths so we can access Octave correctly via the script:

$ octave-wrapper.sh 

GNU Octave, version 3.6.4
Copyright (C) 2013 John W. Eaton and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type `warranty'.
Octave was configured for "x86_64-unknown-linux-gnu".
Additional information about Octave is available at http://www.octave.org.
Please contribute if you find this software useful.
For more information, visit http://www.octave.org/get-involved.html
Read http://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type `news'.
warning: X11 DISPLAY environment variable not set
warning: readline is not linked, so history control is not available
octave:1> 1+1
ans =  2
octave:2> quit()
$

Running Octave code

Now lets run the create our Octave script:

file: ex1_matrix.octave
A = rand(40, 40)
B = A' * A
[v, d] = eig(A)
diag(d)

Run the script:

$ octave ex1_matrix.octave

This should run fairly quickly.

Building the HTCondor job

Although the previous script ran quickly, suppose we needed to use a 100x100 matrix instead of a 10x10 matrix or do this a few thousand times. Since each invocation is independent of others, we can use condor to easily parallelize this and run it on OSG Connect.
First, we'll need to create a wrapper script to setup the environment for Octave and before running it.

file: octave-wrapper.sh
#!/bin/bash
module load libgfortran
module load fftw
module load atlas
module load lapack
module load hdf5
module load qhull
module load pcre
module load SparseSuite
module load glpk
module load octave
  octave $1

 
And of course let's make these wrapper executable

$ chmod +x octave-wrapper.sh

 
Since our previous example only ran for less than a second, let's create a another significantly longer job by operating on a 1000x1000 matrix.

file: ex2_matrix.octave
A = rand(1000,1000)
B = A' * A
[v, d] = eig(A)
diag(d)

 
Now that we've created a wrapper and a batch job, let's build an HTCondor submit file around it.

file: octave.submit
universe = vanilla
log = log/octave.log.$(Cluster)
error = log/octave.err.$(Cluster).$(Process)
output = log/octave.out.$(Cluster).$(Process)
 
# Setup Octave path, run the ex2_matrix.octave script
executable = octave-wrapper.sh
transfer_input_files = ex2_matrix.octave
arguments = ex2_matrix.octave
+ProjectName="ConnectTrain"
 
requirements = (HAS_CVMFS_oasis_opensciencegrid_org =?= TRUE)
queue 100

Since we're using Octave from CVMFS, we will always need to have a requirement (HAS_CVMFS =?= TRUE) that selects nodes with CVMFS installed.

Submit and analyze

Finally, submit the job to OSG Connect!

$ condor_submit octave.submit

After a short wait, the output from the jobs will be in the ~/octave_tutorial/log/ directory.

  • No labels