Department of Electrical and Computer Engineering
Information Infrastructure Institute

Submitting jobs

Partitions and Scheduling

CyBlue has 1024 nodes dynamically allocated into 32 node blocks.  A single partition can be 1024, 512, 128, or 32 nodes. (this is a hardware limitation). Partitions cannot be combined to form a larger partition, eg. you cannot allocate two 32 node partitions for a single 64 node job.  Instead a block of size 128 nodes will be allocated for a 64 node submission.  Jobs must run using 32, 128, 512, or 1024 CPUs.

Availability of queues and partitions is dependent on the time and weekday. The week is broken up into 'development' and 'production' time. Development time is from 10am to 3pm on every day except for the weekend. During development time, the supercomputer will save 128 nodes for testing purposes.  This allows many users to run and test their applications. After 3pm, the entire bluegene will be available for jobs with a maximum job time of 6 hours.   If more time is needed to run a job, please contact cyblue-adm@iastate.edu to request a special time.

Note that the times above are enforced at the time of job submission. You cannot let a job submitted during development time 'sit' in the queue and wait for production time to be executed. If you would like to schedule a production job during development time, you will either need to wait until production time, or use the UNIX 'at' command to schedule the job submission for a later time. Running scripts to continually poll the queue for availability is not permitted, please do not attempt this.



Notes


Submitting a job


To submit a job to run on CyBlue you must use the LSF scheduler. The scheduler uses the following commands:

submit: Submit a job to the scheduler.
delete: Delete a job in the queue.
status: Show queue status.

If you have a job that handles large quantities of file I/O, run your job from the /gpfs01 or /gpfs02 directories! Access by the Blue Gene to /gpfs01 and /gpfs02 is distributed between 8 servers for maximum file I/O bandwidth. You should make a sub-folder for your files, do not place your programs in the root of /gpfs01 or /gpfs02. We suggest creating a directory with your username, eg. /gpfs02/username/project1.

Example

Use of the 'mpirun' command without prior permission is strictly prohibited.