The "fluid" Compute Nodes

attachment:pengsmall.jpg The institute has four compute nodes with 2 Intel Nehalem Quad-Core processors (X5550, 2.66 GHz) and 24 GB memory each. The nodes are integrated into the Phoenix2 cluster with node names n233 - n236. There is a special queue "fluid" that sends jobs to these nodes. The software installed is identical to that of Phoenix2.

Access

For running jobs on the fluid nodes you need an account on phoenix. You can either apply for a regular account on phoenix via the online account managment. Alternatively, if you only want to use the fluid nodes please contact Ernst Haunschmid from ZID or send an e-mail to Ch. Lechner (I will forward it then).

Log into phoenix2 with

ssh -l username phoenix2.zserv.tuwien.ac.at

Compiling and Running

Things work as follows: when you log into phoenix2 you find yourself on the head-node n101. Here you can compile your code and can do (very very small) test runs to see whether the code works. Every application that needs more (than very very small) ressources should be sent to the fluid nodes.

Sending a job to the fluid nodes is done via the Grid Engine. The Grid Engine keeps a job waiting in a queue and automatically starts it as soon as there are enough free processors. In order to do so the Grid Engine has to know the name of the queue, the number of processors needed, the command to start the executable etc. These things are specified in a shell script (see examples below). The job then can be submitted to a queue (-q fluid) with the command qsub:

qsub -q fluid jobscript.sh

(where jobscript.sh is the name of the shell script).

For a description of compiling and running on Phoenix2 please see the Phoenix webpage.

Examples of scripts

Serial

#!/bin/sh
#
# queue
#$ -q fluid

# shell
#$ -S /bin/bash
#
#$ -V

# change into data directory
cd $HOME/path_to_data_directory/

# start serial code
$HOME/path_to_executable/example_exe

I guess when specifying the queue in the script as above the -q fluid option can be omitted when submitting the job.

Parallel with smp

#!/bin/sh
#
#
# queue
#$ -q fluid

# shell
#$ -S /bin/bash

# number of processors
#$ -pe smp 2
#
#$ -V

# cd to data directory
cd $HOME/path_to_data_directory/

# start a multithreaded code with 2 threads
export MP_NUM_THREADS=2
$HOME/path_to_executable/example_threaded_exe

smp means shared memory, so it will only work on a single node (up to 8 threads).

Parallel with MPI

#!/bin/sh
#
#
# queue
#$ -q fluid

# shell
#$ -S /bin/bash

# number of processors
#$ -pe mpi 16
#
#$ -V

# cd to data directory
cd $HOME/path_to_data_directory/

# start MPI code
mpirun $HOME/path_to_exectutable/example_mpi_exe
#
# note: we have tight integration, so no "-np" and no "-hostfile" option for mpirun

(I don't understand the last line, if I use the -np option nothing bad happens.)

In Case of Problems

Please let me know, if something does not work (Ch. Lechner).